[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:20 dist-test-slave-3jw2 krb5kdc[7571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:20 dist-test-slave-3jw2 krb5kdc[7571](info): set up 2 sockets
Sep 26 12:28:20 dist-test-slave-3jw2 krb5kdc[7571](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 12:28:21.869421 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:21 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:21.991797 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40493
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:22.100771 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:22.101034 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:22.101074 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:22.105059 7584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:22.105108 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:22.105126 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:22.105142 7584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:22.105157 7584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:22.110173 7584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:22.111123 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:22.112129 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:22.120574 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.120603 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.120668 7584 server_base.cc:1034] running on GCE node
W20260926 12:28:22.120610 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.121292 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:22.122340 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:22.123522 7584 hybrid_clock.cc:648] HybridClock initialized: now 1790425702123509 us; error 24 us; skew 500 ppm
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:22.129473 7584 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:22.130430 7584 webserver.cc:460] Webserver started at http://127.7.99.62:43567/ using document root <none> and password file <none>
I20260926 12:28:22.131001 7584 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:22.131084 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:22.131299 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:22.133334 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "3c8f91db82654711880db1e9e1cad9b3"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "5a96758b40e139e553136b78373a84ec"
server_key_iv: "08833fbe8905cc1d8ae70d12219638b8"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.134083 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "3c8f91db82654711880db1e9e1cad9b3"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "5a96758b40e139e553136b78373a84ec"
server_key_iv: "08833fbe8905cc1d8ae70d12219638b8"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.137837 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:28:22.140018 7599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.141188 7584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:22.141326 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "3c8f91db82654711880db1e9e1cad9b3"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "5a96758b40e139e553136b78373a84ec"
server_key_iv: "08833fbe8905cc1d8ae70d12219638b8"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.141443 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:22.167021 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:22.170385 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:22.170596 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:22.177654 7584 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:40493
I20260926 12:28:22.177677 7651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:40493 every 8 connection(s)
I20260926 12:28:22.178622 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:22.181318 7652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:22.186758 7652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3: Bootstrap starting.
I20260926 12:28:22.187381 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 7584
I20260926 12:28:22.187564 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:22.188851 7564 external_mini_cluster.cc:1428] Setting key 70bc5fa16acb13cf793941521d10aec6
I20260926 12:28:22.189224 7652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:22.189910 7652 log.cc:824] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:22.191879 7652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3: No bootstrap required, opened a new log
I20260926 12:28:22.196017 7652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } }
I20260926 12:28:22.196338 7652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:22.196403 7652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f91db82654711880db1e9e1cad9b3, State: Initialized, Role: FOLLOWER
I20260926 12:28:22.196866 7652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } }
I20260926 12:28:22.197021 7652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:22.197126 7652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:22.197224 7652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:22.198181 7652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } }
I20260926 12:28:22.198515 7652 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8f91db82654711880db1e9e1cad9b3; no voters:
I20260926 12:28:22.198793 7652 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:22.198890 7657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:22.199085 7657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8f91db82654711880db1e9e1cad9b3, State: Running, Role: LEADER
I20260926 12:28:22.199452 7657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } }
I20260926 12:28:22.200026 7652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:22.201699 7658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8f91db82654711880db1e9e1cad9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } } }
I20260926 12:28:22.201857 7658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:22.202193 7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8f91db82654711880db1e9e1cad9b3. Latest consensus state: current_term: 1 leader_uuid: "3c8f91db82654711880db1e9e1cad9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f91db82654711880db1e9e1cad9b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40493 } } }
I20260926 12:28:22.202356 7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:22.202438 7671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:22.205682 7671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:22.212226 7671 catalog_manager.cc:1347] Generated new cluster ID: 6d730a3f8dc14b3fb3e90d9454dfad15
I20260926 12:28:22.212286 7671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:22.215446 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.191835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.7.99.62:40493)
0926 12:28:22.192302 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.192320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.193565 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 12:28:22.196120 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.196138 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.196568 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.196914 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.199565 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.201058 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.202838 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.203322 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.203446 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.208974 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.208988 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.209003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.209034 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.212039 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.212721 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.212727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.212733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.212871 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.213240 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.213249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.213250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.213631 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.213862 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.214372 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.214641 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260926 12:28:22.234687 7671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:22.235939 7671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:22.246825 7671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8f91db82654711880db1e9e1cad9b3: Generated new TSK 0
I20260926 12:28:22.247675 7671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:22.279237 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:22.386801 7678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:22.387053 7678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:22.387120 7678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:22.390985 7678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:22.391060 7678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:22.391155 7678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:22.396219 7678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:22.397279 7678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:22.398339 7678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:22.407799 7683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.407799 7686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.407852 7684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.408001 7678 server_base.cc:1034] running on GCE node
I20260926 12:28:22.408514 7678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:22.409185 7678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:22.410357 7678 hybrid_clock.cc:648] HybridClock initialized: now 1790425702410336 us; error 34 us; skew 500 ppm
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:22.413053 7678 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:22.413936 7678 webserver.cc:460] Webserver started at http://127.7.99.1:43663/ using document root <none> and password file <none>
I20260926 12:28:22.414474 7678 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:22.414556 7678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:22.414781 7678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:22.416803 7678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "ccfadd896f28414f9ffeaa91f9846acc"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "b0bf6d8f0469e046600b9092975b453d"
server_key_iv: "f7fd924ade4bb744508775d3f7ac09fb"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.417356 7678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "ccfadd896f28414f9ffeaa91f9846acc"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "b0bf6d8f0469e046600b9092975b453d"
server_key_iv: "f7fd924ade4bb744508775d3f7ac09fb"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.421069 7678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:22.423534 7693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.424708 7678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:22.424827 7678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "ccfadd896f28414f9ffeaa91f9846acc"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "b0bf6d8f0469e046600b9092975b453d"
server_key_iv: "f7fd924ade4bb744508775d3f7ac09fb"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.424925 7678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:22.451493 7678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:22.454819 7678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:22.455017 7678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:22.455586 7678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:22.456420 7678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:22.456487 7678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.456557 7678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:22.456609 7678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.464776 7678 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:34997
I20260926 12:28:22.464839 7806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:34997 every 8 connection(s)
I20260926 12:28:22.465631 7678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:22.475168 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 7678
I20260926 12:28:22.475294 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:22.475524 7564 external_mini_cluster.cc:1428] Setting key 9a9547a52e43ca6c4a21bab8bd716f17
I20260926 12:28:22.479311 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.467357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42643 (local address 127.7.99.62:40493)
0926 12:28:22.467494 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.467498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.468299 (+ 801us) server_negotiation.cc:407] Connection header received
0926 12:28:22.469788 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.469792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.469857 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.469956 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.471054 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.471578 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.472776 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.473013 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.473119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.475540 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.475544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.475546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.475570 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.477207 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.477693 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.477699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.477703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.477793 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.478111 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.478117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.478121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.478352 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.478469 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.479027 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.479147 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:22.480043 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.467709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40493 (local address 127.7.99.1:42643)
0926 12:28:22.468125 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:22.468204 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:22.469580 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:22.470111 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:22.470117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:22.470442 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:22.470917 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.470930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.471697 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.471701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.472653 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.472663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.473152 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.473156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.473318 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.473806 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:22.473821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:22.475325 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:22.477327 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.477331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.477341 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.477599 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.477883 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.477885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.477886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.478024 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.478490 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:22.478494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:22.478753 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 12:28:22.478942 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 12:28:22.479251 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260926 12:28:22.481371 7807 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40493
I20260926 12:28:22.481611 7807 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:22.482081 7807 heartbeater.cc:511] Master 127.7.99.62:40493 requested a full tablet report, sending...
I20260926 12:28:22.483472 7616 ts_manager.cc:194] Registered new tserver with Master: ccfadd896f28414f9ffeaa91f9846acc (127.7.99.1:34997)
I20260926 12:28:22.484647 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:42643
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:22.501358 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:22.610047 7812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:22.610316 7812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:22.610384 7812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:22.614444 7812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:22.614516 7812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:22.614599 7812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:22.619782 7812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:22.620878 7812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:22.621883 7812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:22.631664 7818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.631673 7820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.631788 7812 server_base.cc:1034] running on GCE node
W20260926 12:28:22.631664 7817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.632388 7812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:22.632997 7812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:22.634169 7812 hybrid_clock.cc:648] HybridClock initialized: now 1790425702634153 us; error 34 us; skew 500 ppm
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:22.636780 7812 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:22.637651 7812 webserver.cc:460] Webserver started at http://127.7.99.2:35181/ using document root <none> and password file <none>
I20260926 12:28:22.638139 7812 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:22.638195 7812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:22.638345 7812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:22.640393 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "6c17a819b0e44090918faf3ecc009983"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "dfa65a3f70478ec3907744752ee0719c"
server_key_iv: "d27fe9daa90cc0a55825a179990a0600"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.640880 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "6c17a819b0e44090918faf3ecc009983"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "dfa65a3f70478ec3907744752ee0719c"
server_key_iv: "d27fe9daa90cc0a55825a179990a0600"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.644326 7812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:22.646658 7827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.647773 7812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:22.647882 7812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "6c17a819b0e44090918faf3ecc009983"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "dfa65a3f70478ec3907744752ee0719c"
server_key_iv: "d27fe9daa90cc0a55825a179990a0600"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.647977 7812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:22.657114 7812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:22.659884 7812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:22.660054 7812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:22.660636 7812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:22.661368 7812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:22.661410 7812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.661473 7812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:22.661509 7812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.670150 7812 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:41727
I20260926 12:28:22.670169 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:41727 every 8 connection(s)
I20260926 12:28:22.670989 7812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:22.676741 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 7812
I20260926 12:28:22.676832 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:22.677098 7564 external_mini_cluster.cc:1428] Setting key f58c70155a6da4e9ba5d6e5f04ca5bb6
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:22.685246 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.672774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:33155 (local address 127.7.99.62:40493)
0926 12:28:22.672948 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.672952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.673719 (+ 767us) server_negotiation.cc:407] Connection header received
0926 12:28:22.675054 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.675058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.675112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.675211 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.676334 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.676910 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.678195 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.678506 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.678617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.681377 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.681383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.681390 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.681481 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.683165 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.683646 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.683652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.683656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.683759 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.684073 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.684080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.684083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.684363 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.684492 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.684927 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.685050 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:22.685937 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.673131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40493 (local address 127.7.99.2:33155)
0926 12:28:22.673575 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:22.673616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:22.674842 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:22.675383 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:22.675389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:22.675726 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:22.676195 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.676207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.677029 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.677037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.678073 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.678083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.678651 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.678655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.678816 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.679425 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:22.679441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:22.681218 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:22.683290 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.683294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.683305 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.683544 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.683852 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.683854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.683859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.683982 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.684476 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:22.684481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:22.684700 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 12:28:22.684846 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 12:28:22.685073 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:28:22.686987 7941 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40493
I20260926 12:28:22.687223 7941 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:22.687657 7941 heartbeater.cc:511] Master 127.7.99.62:40493 requested a full tablet report, sending...
I20260926 12:28:22.688663 7616 ts_manager.cc:194] Registered new tserver with Master: 6c17a819b0e44090918faf3ecc009983 (127.7.99.2:41727)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:28:22.689287 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:33155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:22.703195 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:22.813169 7946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:22.813421 7946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:22.813463 7946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:22.817446 7946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:22.817500 7946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:22.817557 7946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:22.822559 7946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:22.823534 7946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:22.824543 7946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:22.834203 7954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.834177 7951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:22.834177 7952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:22.834288 7946 server_base.cc:1034] running on GCE node
I20260926 12:28:22.834837 7946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:22.835489 7946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:22.836658 7946 hybrid_clock.cc:648] HybridClock initialized: now 1790425702836625 us; error 47 us; skew 500 ppm
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:22.839802 7946 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:22.840727 7946 webserver.cc:460] Webserver started at http://127.7.99.3:34277/ using document root <none> and password file <none>
I20260926 12:28:22.841264 7946 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:22.841327 7946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:22.841506 7946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:22.843550 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "4b299df008a2414cb4ed7b35f419162c"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "378854b5d418fa112a30f6653c67cc98"
server_key_iv: "66bcfbab7e02255193976e44c99a8f0a"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.844092 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "4b299df008a2414cb4ed7b35f419162c"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "378854b5d418fa112a30f6653c67cc98"
server_key_iv: "66bcfbab7e02255193976e44c99a8f0a"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.848382 7946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:28:22.850817 7961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.852035 7946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:22.852159 7946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "4b299df008a2414cb4ed7b35f419162c"
format_stamp: "Formatted at 2026-09-26 12:28:22 on dist-test-slave-3jw2"
server_key: "378854b5d418fa112a30f6653c67cc98"
server_key_iv: "66bcfbab7e02255193976e44c99a8f0a"
server_key_version: "encryptionkey@0"
I20260926 12:28:22.852293 7946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:22.867807 7946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:22.870932 7946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:22.871083 7946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:22.871608 7946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:22.872437 7946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:22.872483 7946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.872524 7946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:22.872539 7946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:22.881547 7946 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:43387
I20260926 12:28:22.881565 8074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:43387 every 8 connection(s)
I20260926 12:28:22.882436 7946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:22.889114 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 7946
I20260926 12:28:22.889200 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:22.889441 7564 external_mini_cluster.cc:1428] Setting key 1da27e9ffe32d03b001adc4f164de6b2
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:22.896013 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.884610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35167 (local address 127.7.99.62:40493)
0926 12:28:22.884772 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.884777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.885535 (+ 758us) server_negotiation.cc:407] Connection header received
0926 12:28:22.886849 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.886852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.886900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.886974 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.888075 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.888663 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.890047 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.890328 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.890444 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.893057 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.893061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.893063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.893086 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.894329 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.894770 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.894773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.894774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.894844 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.895110 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.895112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.895113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.895278 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.895351 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.895789 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.895882 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:28:22.896836 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.884935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40493 (local address 127.7.99.3:35167)
0926 12:28:22.885383 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:22.885430 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:22.886650 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:22.887112 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:22.887119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:22.887466 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:22.887955 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.887969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.888772 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.888775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.889880 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:22.889904 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.890506 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.890510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:22.890697 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.891346 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:22.891362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:22.892910 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:22.894425 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.894429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.894441 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.894690 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.894914 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:22.894916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:22.894918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:22.895038 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:22.895356 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:22.895360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:22.895578 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 12:28:22.895736 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 12:28:22.895978 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260926 12:28:22.897919 8075 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40493
I20260926 12:28:22.898193 8075 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:22.898677 8075 heartbeater.cc:511] Master 127.7.99.62:40493 requested a full tablet report, sending...
I20260926 12:28:22.899713 7616 ts_manager.cc:194] Registered new tserver with Master: 4b299df008a2414cb4ed7b35f419162c (127.7.99.3:43387)
I20260926 12:28:22.900408 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:35167
I20260926 12:28:22.903473 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:22.927625 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.918876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.7.99.1:34997)
0926 12:28:22.919182 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.919195 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.919222 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:28:22.919289 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.919294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.919474 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.919638 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.920349 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.921114 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.921939 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.922290 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.922403 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.924705 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.924713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.924724 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.924746 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.926126 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.926488 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.926500 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.926501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.926573 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.926779 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.926781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.926782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.926990 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.927141 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.927351 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.927468 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
W20260926 12:28:22.928591 7721 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43488
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:28:22 dist-test-slave-3jw2 krb5kdc[7571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:22.953115 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:22.944981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.7.99.1:34997)
0926 12:28:22.945131 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:22.945136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:22.945254 (+ 118us) server_negotiation.cc:407] Connection header received
0926 12:28:22.945420 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:22.945423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:22.945464 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:22.945554 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:22.946262 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.946779 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.947648 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:22.947878 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:22.947976 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:22.950302 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:22.950305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:22.950307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:22.950329 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:22.951568 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.952022 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.952024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.952025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.952094 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:22.952450 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:22.952452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:22.952453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:22.952601 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:22.952695 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:22.952899 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 12:28:22.952990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:22.954077 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 7678
I20260926 12:28:22.959646 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 7812
I20260926 12:28:22.966325 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 7946
I20260926 12:28:22.971478 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 7584
2026-09-26T12:28:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2152 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:23 dist-test-slave-3jw2 krb5kdc[8089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:23 dist-test-slave-3jw2 krb5kdc[8089](info): set up 2 sockets
Sep 26 12:28:23 dist-test-slave-3jw2 krb5kdc[8089](info): commencing operation
krb5kdc: starting...
W20260926 12:28:24.003996 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:24.121093 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45675
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:24.227108 8102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:24.227357 8102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:24.227396 8102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:24.231172 8102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:24.231223 8102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:24.231243 8102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:24.231261 8102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:24.231276 8102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:24.236390 8102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:24.237323 8102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:24.238314 8102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:24.246462 8107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.246470 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.246472 8108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:24.246745 8102 server_base.cc:1034] running on GCE node
I20260926 12:28:24.247244 8102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:24.248306 8102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:24.249452 8102 hybrid_clock.cc:648] HybridClock initialized: now 1790425704249427 us; error 40 us; skew 500 ppm
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:24.252050 8102 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:24.252986 8102 webserver.cc:460] Webserver started at http://127.7.99.62:41301/ using document root <none> and password file <none>
I20260926 12:28:24.253517 8102 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:24.253611 8102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:24.253825 8102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:24.255775 8102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "7b79ee7e6ab64edab19032eddfccf667"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c24dee676d2781dddff603f0be3a9eb1"
server_key_iv: "60a5b95d2f3ff5c3945739aca5d2aefa"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.256398 8102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "7b79ee7e6ab64edab19032eddfccf667"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c24dee676d2781dddff603f0be3a9eb1"
server_key_iv: "60a5b95d2f3ff5c3945739aca5d2aefa"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.259857 8102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:28:24.262166 8117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.263273 8102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:24.263401 8102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "7b79ee7e6ab64edab19032eddfccf667"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c24dee676d2781dddff603f0be3a9eb1"
server_key_iv: "60a5b95d2f3ff5c3945739aca5d2aefa"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.263504 8102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:24.297976 8102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:24.300755 8102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:24.300946 8102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:24.308001 8102 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:45675
I20260926 12:28:24.308023 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:45675 every 8 connection(s)
I20260926 12:28:24.308938 8102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:24.311584 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:24.316838 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8102
I20260926 12:28:24.316926 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:24.316973 8170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667: Bootstrap starting.
I20260926 12:28:24.317190 7564 external_mini_cluster.cc:1428] Setting key e867c44d470dabf7f5dc29da9410b49b
I20260926 12:28:24.319567 8170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:24.320463 8170 log.cc:824] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:24.322561 8170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667: No bootstrap required, opened a new log
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:24.326828 8170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } }
I20260926 12:28:24.327121 8170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:24.327190 8170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79ee7e6ab64edab19032eddfccf667, State: Initialized, Role: FOLLOWER
I20260926 12:28:24.327662 8170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } }
I20260926 12:28:24.327842 8170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:24.327955 8170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:24.328069 8170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:24.329038 8170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } }
I20260926 12:28:24.329413 8170 leader_election.cc:302] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b79ee7e6ab64edab19032eddfccf667; no voters:
I20260926 12:28:24.329756 8170 leader_election.cc:288] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:24.329838 8175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:24.330072 8175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79ee7e6ab64edab19032eddfccf667, State: Running, Role: LEADER
I20260926 12:28:24.330391 8175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } }
I20260926 12:28:24.331074 8170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:24.331888 8176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b79ee7e6ab64edab19032eddfccf667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } } }
I20260926 12:28:24.332011 8176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:24.332409 8177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b79ee7e6ab64edab19032eddfccf667. Latest consensus state: current_term: 1 leader_uuid: "7b79ee7e6ab64edab19032eddfccf667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b79ee7e6ab64edab19032eddfccf667" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45675 } } }
I20260926 12:28:24.332466 8184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:24.332557 8177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:24.332747 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:24.318460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.7.99.62:45675)
0926 12:28:24.318871 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:24.318890 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:24.318931 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:28:24.319825 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:24.319879 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:24.320354 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:24.320729 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:24.321504 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.322938 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.323730 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.324464 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.324640 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:24.327295 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:24.327301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:24.327322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:24.327347 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:24.329209 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.329685 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.329689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.329693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.329853 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.330256 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.330260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.330262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.330751 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:24.330912 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:24.331374 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 12:28:24.331694 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260926 12:28:24.335765 8184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:24.340466 8184 catalog_manager.cc:1347] Generated new cluster ID: d24544bfbbcb449eba9400b811aa520c
I20260926 12:28:24.340526 8184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:24.350405 8184 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:24.351621 8184 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:24.358621 8184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b79ee7e6ab64edab19032eddfccf667: Generated new TSK 0
I20260926 12:28:24.359277 8184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:24.390405 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:24.497823 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:24.498116 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:24.498188 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:24.502226 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:24.502305 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:24.502391 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:24.507639 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:24.508751 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:24.509765 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:24.519538 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.519492 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.519510 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:24.519701 8196 server_base.cc:1034] running on GCE node
I20260926 12:28:24.520207 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:24.520854 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:24.522029 8196 hybrid_clock.cc:648] HybridClock initialized: now 1790425704522011 us; error 33 us; skew 500 ppm
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:24.524752 8196 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:24.525521 8196 webserver.cc:460] Webserver started at http://127.7.99.1:35531/ using document root <none> and password file <none>
I20260926 12:28:24.526006 8196 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:24.526072 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:24.526226 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:24.528143 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "1eee707f3d924bb49ed00ce9b8bd112c"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "409c5615a5cc052bd3731b7577c4928f"
server_key_iv: "bde4d359496996bc81162b1d87cb80fd"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.528700 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "1eee707f3d924bb49ed00ce9b8bd112c"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "409c5615a5cc052bd3731b7577c4928f"
server_key_iv: "bde4d359496996bc81162b1d87cb80fd"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.532311 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:24.534554 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.535676 8196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:24.535800 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "1eee707f3d924bb49ed00ce9b8bd112c"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "409c5615a5cc052bd3731b7577c4928f"
server_key_iv: "bde4d359496996bc81162b1d87cb80fd"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.535895 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:24.550056 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:24.553195 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:24.553386 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:24.553942 8196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:24.554710 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:24.554754 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.554818 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:24.554848 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.563114 8196 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:42277
I20260926 12:28:24.563144 8324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:42277 every 8 connection(s)
I20260926 12:28:24.563949 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:24.566083 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8196
I20260926 12:28:24.566191 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:24.566519 7564 external_mini_cluster.cc:1428] Setting key 6ab67c3f8fe62f01f959315f5deeb8a5
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:28:24.578125 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:24.565648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:34533 (local address 127.7.99.62:45675)
0926 12:28:24.565788 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:24.565793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:24.566829 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 12:28:24.568469 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:24.568471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:24.568532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:24.568605 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:24.570070 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.570642 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.571871 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.572110 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.572254 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:24.574587 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:24.574590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:24.574592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:24.574614 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:24.576031 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.576577 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.576582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.576584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.576662 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.576993 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.576997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.576998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.577255 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:24.577346 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:24.577838 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 12:28:24.577961 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:28:24.578922 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:24.566145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45675 (local address 127.7.99.1:34533)
0926 12:28:24.566694 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:24.566734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:24.568260 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:24.568785 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:24.568797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:24.569304 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:24.569917 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:24.569931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.570765 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.570768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:24.571754 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:24.571763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.572296 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.572300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:24.572467 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:24.573032 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:24.573048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:24.574421 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:24.576179 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:24.576185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:24.576200 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:24.576472 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:24.576752 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:24.576755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:24.576757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:24.576890 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:24.577344 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:24.577351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:24.577595 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 12:28:24.577800 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 12:28:24.578121 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260926 12:28:24.580229 8325 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45675
I20260926 12:28:24.580448 8325 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:24.580883 8325 heartbeater.cc:511] Master 127.7.99.62:45675 requested a full tablet report, sending...
I20260926 12:28:24.582388 8134 ts_manager.cc:194] Registered new tserver with Master: 1eee707f3d924bb49ed00ce9b8bd112c (127.7.99.1:42277)
I20260926 12:28:24.584089 8134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:34533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:24.594826 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:24.706135 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:24.706429 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:24.706504 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:24.711037 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:24.711122 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:24.711218 8330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:24.719022 8330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:24.720206 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:24.721272 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:24.731133 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.731258 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.731133 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:24.731364 8330 server_base.cc:1034] running on GCE node
I20260926 12:28:24.731871 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:24.732578 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:24.733727 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790425704733698 us; error 40 us; skew 500 ppm
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:24.736446 8330 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:24.737282 8330 webserver.cc:460] Webserver started at http://127.7.99.2:42419/ using document root <none> and password file <none>
I20260926 12:28:24.737854 8330 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:24.737938 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:24.738186 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:24.740231 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "43d42395ddca45debc1ab58be0f73acc"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c438621ee6995142e37f17ca7dcac0d8"
server_key_iv: "fd7ef5ec8ffc9cb500e5c83a15885c81"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.740796 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "43d42395ddca45debc1ab58be0f73acc"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c438621ee6995142e37f17ca7dcac0d8"
server_key_iv: "fd7ef5ec8ffc9cb500e5c83a15885c81"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.744264 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:28:24.746538 8345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.747817 8330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:24.748056 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "43d42395ddca45debc1ab58be0f73acc"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "c438621ee6995142e37f17ca7dcac0d8"
server_key_iv: "fd7ef5ec8ffc9cb500e5c83a15885c81"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.748222 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:24.767309 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:24.770496 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:24.770699 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:24.771261 8330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:24.772046 8330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:24.772111 8330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.772197 8330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:24.772244 8330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.780421 8330 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:38561
I20260926 12:28:24.780467 8458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:38561 every 8 connection(s)
I20260926 12:28:24.781262 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:24.791152 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8330
I20260926 12:28:24.791280 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:24.791646 7564 external_mini_cluster.cc:1428] Setting key ee124834ccb37b68c9553de057e0eaf2
I20260926 12:28:24.794907 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:24.782896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:52835 (local address 127.7.99.62:45675)
0926 12:28:24.783040 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:24.783044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:24.783803 (+ 759us) server_negotiation.cc:407] Connection header received
0926 12:28:24.785234 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:24.785237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:24.785292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:24.785396 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:24.786507 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.787032 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.788124 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.788398 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.788507 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:24.790987 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:24.790994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:24.790998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:24.791028 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:24.792674 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.793292 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.793301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.793303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.793435 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:24.793794 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:24.793797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:24.793799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:24.793985 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:24.794113 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:24.794687 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 12:28:24.794793 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:28:24.796140 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:24.783248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45675 (local address 127.7.99.2:52835)
0926 12:28:24.783666 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:24.783708 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:24.785028 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:24.785578 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:24.785584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:24.785919 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:24.786362 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:24.786374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.787147 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.787155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:24.787989 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:24.787997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:24.788539 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:24.788543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:24.788702 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:24.789257 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:24.789280 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:24.790824 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:24.792795 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:24.792800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:24.792811 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:24.793187 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:24.793532 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:24.793536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:24.793538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:24.793688 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:24.794109 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:24.794115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:24.794371 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 12:28:24.794613 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 12:28:24.795010 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260926 12:28:24.797314 8459 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45675
I20260926 12:28:24.797580 8459 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:24.798059 8459 heartbeater.cc:511] Master 127.7.99.62:45675 requested a full tablet report, sending...
I20260926 12:28:24.799213 8134 ts_manager.cc:194] Registered new tserver with Master: 43d42395ddca45debc1ab58be0f73acc (127.7.99.2:38561)
I20260926 12:28:24.800261 8134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:52835
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:24.818801 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:24.927806 8464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:24.928084 8464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:24.928176 8464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:24.932374 8464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:24.932451 8464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:24.932542 8464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:24.937753 8464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:24.938768 8464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:24.939790 8464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:24.949039 8469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:24.949026 8472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:24.949235 8464 server_base.cc:1034] running on GCE node
W20260926 12:28:24.949060 8470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:24.949738 8464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:24.950387 8464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:24.951565 8464 hybrid_clock.cc:648] HybridClock initialized: now 1790425704951539 us; error 44 us; skew 500 ppm
Sep 26 12:28:24 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:24.954262 8464 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:24.955135 8464 webserver.cc:460] Webserver started at http://127.7.99.3:41257/ using document root <none> and password file <none>
I20260926 12:28:24.955684 8464 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:24.955770 8464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:24.955983 8464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:24.958033 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "1985b8cb53714d12b39fef03732eda10"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "4840a13844d02b0694d00136f108143e"
server_key_iv: "1d0870a9c272f6f8b951599dbf495199"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.958585 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "1985b8cb53714d12b39fef03732eda10"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "4840a13844d02b0694d00136f108143e"
server_key_iv: "1d0870a9c272f6f8b951599dbf495199"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.962107 8464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:28:24.964375 8479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.965505 8464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:24.965626 8464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "1985b8cb53714d12b39fef03732eda10"
format_stamp: "Formatted at 2026-09-26 12:28:24 on dist-test-slave-3jw2"
server_key: "4840a13844d02b0694d00136f108143e"
server_key_iv: "1d0870a9c272f6f8b951599dbf495199"
server_key_version: "encryptionkey@0"
I20260926 12:28:24.965723 8464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:24.983903 8464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:24.986981 8464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:24.987172 8464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:24.987723 8464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:24.988543 8464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:24.988606 8464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.988670 8464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:24.988715 8464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:24.997817 8464 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33255
I20260926 12:28:24.997848 8592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33255 every 8 connection(s)
I20260926 12:28:24.998670 8464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:25.004096 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8464
I20260926 12:28:25.004264 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:25.004588 7564 external_mini_cluster.cc:1428] Setting key 626a8b126efa012cbefa2b1cdb223e14
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:25.012432 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.000524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:48969 (local address 127.7.99.62:45675)
0926 12:28:25.000652 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.000655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.001504 (+ 849us) server_negotiation.cc:407] Connection header received
0926 12:28:25.002810 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.002813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.002883 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.002960 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:25.004059 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.004836 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.006209 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.006518 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.006621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.009033 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:25.009037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:25.009040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:25.009065 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:25.010489 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.011018 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.011022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.011023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.011101 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.011415 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.011419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.011421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.011592 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:25.011694 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.012140 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.012271 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:25.013173 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.000933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45675 (local address 127.7.99.3:48969)
0926 12:28:25.001354 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.001392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.002614 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.003116 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.003123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.003444 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:25.003915 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.003928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.004953 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.004958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.006094 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.006104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.006651 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.006655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.006816 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.007390 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:25.007405 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:25.008865 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:25.010626 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:25.010641 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:25.010653 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:25.010917 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:25.011193 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:25.011196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:25.011198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:25.011327 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:25.011689 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:25.011695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:25.011924 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.012099 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.012416 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260926 12:28:25.014318 8593 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45675
I20260926 12:28:25.014508 8593 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:25.014899 8593 heartbeater.cc:511] Master 127.7.99.62:45675 requested a full tablet report, sending...
I20260926 12:28:25.015816 8134 ts_manager.cc:194] Registered new tserver with Master: 1985b8cb53714d12b39fef03732eda10 (127.7.99.3:33255)
I20260926 12:28:25.016381 8134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:48969
I20260926 12:28:25.018100 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:28:25.027484 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.021159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.7.99.62:45675)
0926 12:28:25.021263 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.021267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.021413 (+ 146us) server_negotiation.cc:407] Connection header received
0926 12:28:25.021507 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.021509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.021542 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.021607 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:25.022217 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.022686 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.023428 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.023633 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.023708 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.024857 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:25.024860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:25.024862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:25.024882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:25.026221 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.026545 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.026548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.026549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.026624 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.026825 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.026827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.026828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.027029 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:25.027128 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.027291 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.027372 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:28:25.032300 8134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:25.035816 8134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:25.049422 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.043717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:38561 (local address 127.0.0.1:38910)
0926 12:28:25.044050 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.044086 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.044244 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.045011 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.045016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.045033 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:25.045125 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.045129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.046550 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.046554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.048038 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.048047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.048895 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.048899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.049002 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.049040 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.049152 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.049270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":96,"thread_start_us":65,"threads_started":1}
I20260926 12:28:25.049430 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.043955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.7.99.2:38561)
0926 12:28:25.044306 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.044321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.044346 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:28:25.044436 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.044443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.044642 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.044809 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:25.045272 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.046402 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.048128 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.048775 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.048895 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.049049 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.049179 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.049273 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260926 12:28:25.049865 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.044425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33255 (local address 127.0.0.1:40802)
0926 12:28:25.044797 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.044811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.044887 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.045684 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.045688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.045707 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:25.045804 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.045814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.047002 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.047008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.048266 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.048275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.049456 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.049460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.049563 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.049607 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.049673 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.049737 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":309,"thread_start_us":75,"threads_started":1}
I20260926 12:28:25.050278 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.044804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.7.99.3:33255)
0926 12:28:25.045222 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.045234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.045258 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:28:25.045328 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.045333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.045537 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.045952 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:25.046082 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.046887 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.048392 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.049178 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.049279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.049398 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.050040 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.050130 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":129,"threads_started":1}
I20260926 12:28:25.050709 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.044562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:42277 (local address 127.0.0.1:39958)
0926 12:28:25.045770 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.045784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.045874 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.046287 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.046290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.046309 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:25.046387 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.046393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.047724 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.047727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.048965 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.048975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.050356 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.050360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.050447 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.050470 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.050521 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.050583 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":58,"thread_start_us":95,"threads_started":1}
I20260926 12:28:25.051546 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.045026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.7.99.1:42277)
0926 12:28:25.045318 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.045331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.045870 (+ 539us) server_negotiation.cc:407] Connection header received
0926 12:28:25.045978 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.045983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.046154 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.046284 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:25.046484 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.047594 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.049114 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.050247 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.051089 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.051230 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.051278 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.051406 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260926 12:28:25.052327 8393 tablet_service.cc:1467] Processing CreateTablet for tablet 1242b467d0114588bb0dbe8143621470 (DEFAULT_TABLE table=test-table [id=e198abfb72c54de88073fb8bc14b66db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:25.052947 8527 tablet_service.cc:1467] Processing CreateTablet for tablet 1242b467d0114588bb0dbe8143621470 (DEFAULT_TABLE table=test-table [id=e198abfb72c54de88073fb8bc14b66db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:25.053398 8393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1242b467d0114588bb0dbe8143621470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:25.053905 8527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1242b467d0114588bb0dbe8143621470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:25.054003 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 1242b467d0114588bb0dbe8143621470 (DEFAULT_TABLE table=test-table [id=e198abfb72c54de88073fb8bc14b66db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:25.054899 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1242b467d0114588bb0dbe8143621470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:25.058975 8610 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Bootstrap starting.
I20260926 12:28:25.061753 8610 tablet_bootstrap.cc:654] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:25.061781 8612 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: Bootstrap starting.
I20260926 12:28:25.062469 8610 log.cc:824] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:25.063469 8612 tablet_bootstrap.cc:654] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:25.063118 8611 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Bootstrap starting.
I20260926 12:28:25.064419 8610 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: No bootstrap required, opened a new log
I20260926 12:28:25.064602 8610 ts_tablet_manager.cc:1397] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:28:25.064742 8612 log.cc:824] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:25.066200 8611 tablet_bootstrap.cc:654] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:25.066715 8612 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: No bootstrap required, opened a new log
I20260926 12:28:25.066913 8612 ts_tablet_manager.cc:1397] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:25.067340 8611 log.cc:824] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:25.068092 8610 raft_consensus.cc:348] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.068389 8610 raft_consensus.cc:374] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:25.068445 8610 raft_consensus.cc:729] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d42395ddca45debc1ab58be0f73acc, State: Initialized, Role: FOLLOWER
I20260926 12:28:25.068912 8610 consensus_queue.cc:260] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.069273 8611 tablet_bootstrap.cc:492] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: No bootstrap required, opened a new log
I20260926 12:28:25.069576 8459 heartbeater.cc:503] Master 127.7.99.62:45675 was elected leader, sending a full tablet report...
I20260926 12:28:25.069833 8610 ts_tablet_manager.cc:1428] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:25.070091 8611 ts_tablet_manager.cc:1397] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:28:25.070565 8612 raft_consensus.cc:348] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.070835 8612 raft_consensus.cc:374] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:25.070894 8612 raft_consensus.cc:729] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eee707f3d924bb49ed00ce9b8bd112c, State: Initialized, Role: FOLLOWER
I20260926 12:28:25.071277 8612 consensus_queue.cc:260] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.071870 8325 heartbeater.cc:503] Master 127.7.99.62:45675 was elected leader, sending a full tablet report...
I20260926 12:28:25.072225 8612 ts_tablet_manager.cc:1428] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 12:28:25.073451 8611 raft_consensus.cc:348] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.073710 8611 raft_consensus.cc:374] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:25.073770 8611 raft_consensus.cc:729] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1985b8cb53714d12b39fef03732eda10, State: Initialized, Role: FOLLOWER
I20260926 12:28:25.074178 8611 consensus_queue.cc:260] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.074757 8593 heartbeater.cc:503] Master 127.7.99.62:45675 was elected leader, sending a full tablet report...
I20260926 12:28:25.075028 8611 ts_tablet_manager.cc:1428] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:25.186039 8616 raft_consensus.cc:482] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:25.186297 8616 raft_consensus.cc:504] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.187446 8616 leader_election.cc:288] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1985b8cb53714d12b39fef03732eda10 (127.7.99.3:33255), 1eee707f3d924bb49ed00ce9b8bd112c (127.7.99.1:42277)
I20260926 12:28:25.190972 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.187780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:37851 (local address 127.7.99.3:33255)
0926 12:28:25.187942 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.187947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.187965 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:28:25.188103 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.188106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.188146 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.188252 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:25.188527 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.189010 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190003 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.190420 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190552 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.190614 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.190790 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.190854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 12:28:25.190984 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.187718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33255 (local address 127.7.99.2:37851)
0926 12:28:25.187844 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.187860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.187955 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.188263 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.188266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.188306 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:25.188414 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.188419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.189113 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.189118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.189874 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.189882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.190552 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.190673 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.190690 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.190775 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.190842 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1,"spinlock_wait_cycles":27264}
I20260926 12:28:25.191183 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.187915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:39341 (local address 127.7.99.1:42277)
0926 12:28:25.188031 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.188036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.188266 (+ 230us) server_negotiation.cc:407] Connection header received
0926 12:28:25.188432 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.188435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.188476 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.188554 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:25.188801 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.189255 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190236 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.190665 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.190837 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.191004 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.191067 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:25.191203 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.187722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:42277 (local address 127.7.99.2:39341)
0926 12:28:25.188175 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:25.188185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:25.188323 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:25.188564 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:25.188566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:25.188580 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:25.188699 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.188705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.189354 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.189360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.190122 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:25.190130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.190778 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.190782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:25.190884 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.190909 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:28:25.190988 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:28:25.191052 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":103,"spinlock_wait_cycles":896,"thread_start_us":141,"threads_started":1}
I20260926 12:28:25.191680 8547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1242b467d0114588bb0dbe8143621470" candidate_uuid: "43d42395ddca45debc1ab58be0f73acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1985b8cb53714d12b39fef03732eda10" is_pre_election: true
I20260926 12:28:25.191783 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1242b467d0114588bb0dbe8143621470" candidate_uuid: "43d42395ddca45debc1ab58be0f73acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" is_pre_election: true
I20260926 12:28:25.192001 8547 raft_consensus.cc:2454] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43d42395ddca45debc1ab58be0f73acc in term 0.
I20260926 12:28:25.192075 8279 raft_consensus.cc:2454] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43d42395ddca45debc1ab58be0f73acc in term 0.
I20260926 12:28:25.192544 8348 leader_election.cc:302] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1985b8cb53714d12b39fef03732eda10, 1eee707f3d924bb49ed00ce9b8bd112c, 43d42395ddca45debc1ab58be0f73acc; no voters:
I20260926 12:28:25.192806 8616 raft_consensus.cc:2781] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:25.192883 8616 raft_consensus.cc:482] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:25.192927 8616 raft_consensus.cc:3037] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:25.193958 8616 raft_consensus.cc:504] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.194303 8616 leader_election.cc:288] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [CANDIDATE]: Term 1 election: Requested vote from peers 1985b8cb53714d12b39fef03732eda10 (127.7.99.3:33255), 1eee707f3d924bb49ed00ce9b8bd112c (127.7.99.1:42277)
I20260926 12:28:25.194621 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1242b467d0114588bb0dbe8143621470" candidate_uuid: "43d42395ddca45debc1ab58be0f73acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eee707f3d924bb49ed00ce9b8bd112c"
I20260926 12:28:25.194621 8547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1242b467d0114588bb0dbe8143621470" candidate_uuid: "43d42395ddca45debc1ab58be0f73acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1985b8cb53714d12b39fef03732eda10"
I20260926 12:28:25.194734 8547 raft_consensus.cc:3037] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:25.194736 8279 raft_consensus.cc:3037] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:25.195935 8279 raft_consensus.cc:2454] T 1242b467d0114588bb0dbe8143621470 P 1eee707f3d924bb49ed00ce9b8bd112c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43d42395ddca45debc1ab58be0f73acc in term 1.
I20260926 12:28:25.195935 8547 raft_consensus.cc:2454] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43d42395ddca45debc1ab58be0f73acc in term 1.
I20260926 12:28:25.196252 8346 leader_election.cc:302] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1985b8cb53714d12b39fef03732eda10, 1eee707f3d924bb49ed00ce9b8bd112c, 43d42395ddca45debc1ab58be0f73acc; no voters:
I20260926 12:28:25.196388 8616 raft_consensus.cc:2781] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:25.196573 8616 raft_consensus.cc:686] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [term 1 LEADER]: Becoming Leader. State: Replica: 43d42395ddca45debc1ab58be0f73acc, State: Running, Role: LEADER
I20260926 12:28:25.196882 8616 consensus_queue.cc:237] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } }
I20260926 12:28:25.199716 8134 catalog_manager.cc:5495] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc reported cstate change: term changed from 0 to 1, leader changed from <none> to 43d42395ddca45debc1ab58be0f73acc (127.7.99.2). New cstate: current_term: 1 leader_uuid: "43d42395ddca45debc1ab58be0f73acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d42395ddca45debc1ab58be0f73acc" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1985b8cb53714d12b39fef03732eda10" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eee707f3d924bb49ed00ce9b8bd112c" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 42277 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:25.247542 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.238836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.7.99.1:42277)
0926 12:28:25.238990 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.238995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.239132 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:28:25.239238 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.239240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.239284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.239380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:25.240227 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.240774 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.241573 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.241772 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.241842 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.244328 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:25.244339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:25.244355 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:25.244376 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:25.245931 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.246345 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.246350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.246352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.246437 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.246731 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.246733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.246734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.246956 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:25.247114 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.247286 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.247378 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 12:28:25.249635 8594 tablet.cc:2369] T 1242b467d0114588bb0dbe8143621470 P 1985b8cb53714d12b39fef03732eda10: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425705, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425705, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:25.274309 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.265730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.7.99.1:42277)
0926 12:28:25.265909 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.265913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.265983 (+ 70us) server_negotiation.cc:407] Connection header received
0926 12:28:25.266107 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.266110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.266153 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.266222 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:25.266919 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.267427 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.268685 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.268934 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.269010 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.271509 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:25.271512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:25.271514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:25.271535 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:25.272863 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.273268 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.273270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.273271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.273337 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.273558 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.273560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.273561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.273702 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:25.273792 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.274014 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.274110 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 12:28:25.282385 8460 tablet.cc:2369] T 1242b467d0114588bb0dbe8143621470 P 43d42395ddca45debc1ab58be0f73acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:25.299124 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:25.290735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.7.99.1:42277)
0926 12:28:25.290919 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:25.290924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:25.291052 (+ 128us) server_negotiation.cc:407] Connection header received
0926 12:28:25.291215 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:25.291217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:25.291257 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:25.291343 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:25.292016 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.292609 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.293472 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:25.293686 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:25.293786 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:25.296064 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:25.296073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:25.296075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:25.296096 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:25.297530 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.297947 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.297950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.297952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.298047 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:25.298308 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:25.298312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:25.298314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:25.298479 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:25.298605 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:25.298840 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 12:28:25.298957 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 12:28:25.300457 8239 tablet_service.cc:2908] Rejecting scan request for tablet 1242b467d0114588bb0dbe8143621470: Uninitialized: clean time has not yet been initialized
I20260926 12:28:25.302404 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8196
I20260926 12:28:25.308712 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8330
I20260926 12:28:25.314296 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8464
I20260926 12:28:25.320803 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8102
2026-09-26T12:28:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2348 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8634](info): set up 2 sockets
Sep 26 12:28:25 dist-test-slave-3jw2 krb5kdc[8634](info): commencing operation
krb5kdc: starting...
W20260926 12:28:26.356101 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:26 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:26.474977 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40929
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:40929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:26.601189 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:26.601480 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:26.601553 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:26.605682 8647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:26.605754 8647 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:28:26.605784 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:26.605826 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:26.605861 8647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:26.605895 8647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:26.611193 8647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:40929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:26.612303 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:26.613302 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:26.621527 8652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:26.621500 8655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:26.621495 8653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:26.622272 8647 server_base.cc:1034] running on GCE node
I20260926 12:28:26.622792 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:26.623832 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:26.625011 8647 hybrid_clock.cc:648] HybridClock initialized: now 1790425706624991 us; error 41 us; skew 500 ppm
Sep 26 12:28:26 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:26.627609 8647 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:26.628428 8647 webserver.cc:460] Webserver started at http://127.7.99.62:44123/ using document root <none> and password file <none>
I20260926 12:28:26.628922 8647 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:26.628976 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:26.629132 8647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:26.631107 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "c84637f274484d2c9bb3364886bfb51c"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "58bab3a29a3865bfd17b6ae2db9deabf"
server_key_iv: "7c4dec51a129ae23fa39c5344b7d23e8"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.631594 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "c84637f274484d2c9bb3364886bfb51c"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "58bab3a29a3865bfd17b6ae2db9deabf"
server_key_iv: "7c4dec51a129ae23fa39c5344b7d23e8"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.634994 8647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:28:26.637266 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:26.638393 8647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:26.638484 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "c84637f274484d2c9bb3364886bfb51c"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "58bab3a29a3865bfd17b6ae2db9deabf"
server_key_iv: "7c4dec51a129ae23fa39c5344b7d23e8"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.638578 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:26.655402 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:26.658185 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:26.658332 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:26.665724 8647 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:40929
I20260926 12:28:26.665752 8724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:40929 every 8 connection(s)
I20260926 12:28:26.666649 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:26.669348 8725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:26.670892 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8647
I20260926 12:28:26.670998 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:26.671275 7564 external_mini_cluster.cc:1428] Setting key 72909988b0124f95fb5140c8f1b7c095
I20260926 12:28:26.675314 8725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Bootstrap starting.
I20260926 12:28:26.677717 8725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:26.678732 8725 log.cc:824] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:28:26 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:26.680676 8725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: No bootstrap required, opened a new log
I20260926 12:28:26.684080 8725 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } }
I20260926 12:28:26.684377 8725 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:26.684458 8725 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84637f274484d2c9bb3364886bfb51c, State: Initialized, Role: FOLLOWER
I20260926 12:28:26.684885 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:26.672696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.7.99.62:40929)
0926 12:28:26.673123 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:26.673137 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:26.673190 (+ 53us) server_negotiation.cc:407] Connection header received
0926 12:28:26.673785 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:26.673800 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:26.674121 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:26.674449 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:26.675217 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.676370 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.677229 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.677709 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.677875 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:26.680398 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:26.680403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:26.680416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:26.680440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:26.682063 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.682490 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.682494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.682499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.682607 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.682885 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.682889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.682891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.683258 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:26.683468 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:26.683714 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:28:26.683978 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260926 12:28:26.684909 8725 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } }
I20260926 12:28:26.685122 8725 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:26.685194 8725 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:26.685292 8725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:26.686316 8725 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } }
I20260926 12:28:26.686641 8725 leader_election.cc:302] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84637f274484d2c9bb3364886bfb51c; no voters:
I20260926 12:28:26.686971 8725 leader_election.cc:288] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:26.687098 8730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:26.687328 8730 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [term 1 LEADER]: Becoming Leader. State: Replica: c84637f274484d2c9bb3364886bfb51c, State: Running, Role: LEADER
I20260926 12:28:26.687675 8730 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } }
I20260926 12:28:26.688196 8725 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:26.689350 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84637f274484d2c9bb3364886bfb51c. Latest consensus state: current_term: 1 leader_uuid: "c84637f274484d2c9bb3364886bfb51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } } }
I20260926 12:28:26.689366 8731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84637f274484d2c9bb3364886bfb51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84637f274484d2c9bb3364886bfb51c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40929 } } }
I20260926 12:28:26.689505 8731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:26.689504 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:26.689882 8739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:26.693001 8739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:26.698273 8739 catalog_manager.cc:1347] Generated new cluster ID: 02a915115e794c3bb3800c81e133af97
I20260926 12:28:26.698342 8739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:26.705296 8739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:26.706178 8739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:26.711764 8739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Generated new TSK 0
I20260926 12:28:26.712550 8739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:26.743673 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40929
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:26.854417 8751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:26.854715 8751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:26.854791 8751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:26.858830 8751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:26.858913 8751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:26.858983 8751 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:26.859071 8751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:26.864421 8751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:26.865458 8751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:26.866499 8751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:26.876413 8757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:26.876431 8751 server_base.cc:1034] running on GCE node
W20260926 12:28:26.876367 8756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:26.876382 8759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:26.876958 8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:26.877597 8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:26.878756 8751 hybrid_clock.cc:648] HybridClock initialized: now 1790425706878723 us; error 46 us; skew 500 ppm
Sep 26 12:28:26 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:26.881480 8751 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:26.882323 8751 webserver.cc:460] Webserver started at http://127.7.99.1:40823/ using document root <none> and password file <none>
I20260926 12:28:26.882885 8751 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:26.882972 8751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:26.883203 8751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:26.885313 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "f49e2f150b554f4b9e7c7b9f46330562"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "a6eed534fba25fbf6a874df87a7adbab"
server_key_iv: "cdce55dfd693d77ca9ee030bdba0c406"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.885910 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "f49e2f150b554f4b9e7c7b9f46330562"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "a6eed534fba25fbf6a874df87a7adbab"
server_key_iv: "cdce55dfd693d77ca9ee030bdba0c406"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.889854 8751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:28:26.892331 8766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:26.893514 8751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:26.893662 8751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "f49e2f150b554f4b9e7c7b9f46330562"
format_stamp: "Formatted at 2026-09-26 12:28:26 on dist-test-slave-3jw2"
server_key: "a6eed534fba25fbf6a874df87a7adbab"
server_key_iv: "cdce55dfd693d77ca9ee030bdba0c406"
server_key_version: "encryptionkey@0"
I20260926 12:28:26.893774 8751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:26.904562 8751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:26.907200 8751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:26.907400 8751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:26.909018 8751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:26.909093 8751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:26.909160 8751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:26.909209 8751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:26.919526 8751 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:46787
Sep 26 12:28:26 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:26.919577 8881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:46787 every 8 connection(s)
I20260926 12:28:26.920622 8751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:26.923487 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:26.910431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:60723 (local address 127.7.99.62:40929)
0926 12:28:26.910593 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:26.910598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:26.911373 (+ 775us) server_negotiation.cc:407] Connection header received
0926 12:28:26.912700 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:26.912704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:26.912760 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:26.912839 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:26.914385 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.915308 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.916906 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.917308 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.917430 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:26.920278 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:26.920282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:26.920285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:26.920312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:26.921837 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.922276 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.922278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.922279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.922352 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.922620 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.922622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.922623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.922775 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:26.922843 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:26.923274 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 12:28:26.923369 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:28:26.924309 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:26.910722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.1:60723)
0926 12:28:26.911231 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:26.911279 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:26.912451 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:26.913007 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:26.913016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:26.913503 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:26.914205 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:26.914226 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.915461 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.915464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:26.916772 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:26.916782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.917486 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.917491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:26.917713 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:26.918368 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:26.918387 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:26.920086 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:26.921933 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:26.921937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:26.921947 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:26.922180 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:26.922426 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:26.922428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:26.922430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:26.922546 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:26.922859 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:26.922864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:26.923072 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 12:28:26.923215 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 12:28:26.923440 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":187,"threads_started":1}
I20260926 12:28:26.925444 8882 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40929
I20260926 12:28:26.925585 8882 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:26.926039 8882 heartbeater.cc:511] Master 127.7.99.62:40929 requested a full tablet report, sending...
I20260926 12:28:26.927604 8678 ts_manager.cc:194] Registered new tserver with Master: f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787)
I20260926 12:28:26.928939 8678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:60723
I20260926 12:28:26.930068 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8751
I20260926 12:28:26.930190 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:26.930444 7564 external_mini_cluster.cc:1428] Setting key 8cc4ff1ed188759540ad67d25050f181
I20260926 12:28:26.934701 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:26.927554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:59891 (local address 127.7.99.62:40929)
0926 12:28:26.927661 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:26.927670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:26.927791 (+ 121us) server_negotiation.cc:407] Connection header received
0926 12:28:26.927907 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:26.927909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:26.927945 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:26.928005 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:26.928471 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.928948 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.929874 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.930188 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.930290 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:26.931586 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:26.931590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:26.931593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:26.931614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:26.933164 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.933552 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.933557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.933560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.933650 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:26.933895 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:26.933899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:26.933901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:26.934122 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:26.934230 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:26.934448 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 12:28:26.934550 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260926 12:28:26.934708 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:26.927426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.1:59891)
0926 12:28:26.927721 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:26.927735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:26.927827 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:26.928043 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:26.928045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:26.928260 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:26.928373 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:26.928380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.929071 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.929074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:26.929764 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:26.929773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:26.930395 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:26.930398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:26.930479 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:26.931056 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:26.931070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:26.931455 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:26.933269 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:26.933272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:26.933273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:26.933453 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:26.933740 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:26.933742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:26.933743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:26.933812 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:26.934217 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:26.934219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:26.934285 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 12:28:26.934392 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 12:28:26.934517 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:26.957156 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40929
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:27.065864 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:27.066114 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:27.066190 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:27.070197 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:27.070248 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:27.070320 8891 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:27.070391 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:27.075577 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:27.076597 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:27.077579 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:27.087091 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:27.087106 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:27.087106 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:27.087248 8891 server_base.cc:1034] running on GCE node
I20260926 12:28:27.087809 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:27.088568 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:27.089736 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790425707089702 us; error 56 us; skew 500 ppm
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:27.092547 8891 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:27.093261 8891 webserver.cc:460] Webserver started at http://127.7.99.2:33841/ using document root <none> and password file <none>
I20260926 12:28:27.093739 8891 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:27.093791 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:27.093938 8891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:27.095788 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "cebd9fe5929641b6bb4e3b65a563e953"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "2a71ccc92da679671009535936111556"
server_key_iv: "2e286c6aad46a421b7ab2e5fd968c7f4"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.096341 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "cebd9fe5929641b6bb4e3b65a563e953"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "2a71ccc92da679671009535936111556"
server_key_iv: "2e286c6aad46a421b7ab2e5fd968c7f4"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.099810 8891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:27.102138 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:27.103305 8891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:27.103431 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "cebd9fe5929641b6bb4e3b65a563e953"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "2a71ccc92da679671009535936111556"
server_key_iv: "2e286c6aad46a421b7ab2e5fd968c7f4"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.103543 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:27.137601 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:27.140919 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:27.141146 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:27.142729 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:27.142799 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:27.142870 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:27.142913 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:27.153930 8891 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:32919
I20260926 12:28:27.154026 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:32919 every 8 connection(s)
I20260926 12:28:27.155007 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:27.156324 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.143666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:42225 (local address 127.7.99.62:40929)
0926 12:28:27.143797 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.143801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.144915 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 12:28:27.145972 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.145975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.146034 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.146134 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.147712 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.148317 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.149440 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.149739 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.149862 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.152665 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.152667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.152670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.152692 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.154254 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.154786 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.154788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.154790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.154856 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.155207 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.155209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.155211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.155403 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.155497 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.156020 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.156123 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:28:27.157037 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.144275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.2:42225)
0926 12:28:27.144757 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.144798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.145747 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.146359 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.146370 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.146901 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:27.147564 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.147584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.148431 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.148435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.149318 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.149326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.149901 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.149905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.150099 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.150687 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:27.150711 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:27.152457 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:27.154353 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.154359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.154370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.154691 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.154942 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.154947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.154950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.155123 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.155546 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:27.155551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:27.155810 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.155967 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.156255 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260926 12:28:27.158131 9022 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40929
I20260926 12:28:27.158252 9022 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:27.158741 9022 heartbeater.cc:511] Master 127.7.99.62:40929 requested a full tablet report, sending...
I20260926 12:28:27.159746 8679 ts_manager.cc:194] Registered new tserver with Master: cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2:32919)
I20260926 12:28:27.160549 8679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:42225
I20260926 12:28:27.162884 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 8891
I20260926 12:28:27.163029 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:27.163338 7564 external_mini_cluster.cc:1428] Setting key 005be6e3078c534d3a2379731c3b3f7c
I20260926 12:28:27.168497 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.160335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:33255 (local address 127.7.99.62:40929)
0926 12:28:27.160452 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.160456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.161028 (+ 572us) server_negotiation.cc:407] Connection header received
0926 12:28:27.161144 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.161147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.161189 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.161269 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.161665 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.162367 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.163297 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.163579 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.163686 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.164943 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.164946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.164949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.164972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.166673 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.167298 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.167300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.167301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.167369 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.167683 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.167685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.167686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.167868 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.167940 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.168295 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.168380 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:28:27.168715 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.160282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.2:33255)
0926 12:28:27.160934 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.160948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.161052 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.161289 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.161291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.161466 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:27.161561 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.161566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.162490 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.162492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.163153 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.163168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.163688 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.163692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.163779 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.164359 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:27.164377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:27.164813 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:27.166824 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.166831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.166833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.167181 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.167458 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.167462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.167466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.167573 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.167965 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:27.167971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:27.168086 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.168247 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.168418 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:27.189389 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:40929
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:27.297030 9031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:27.297260 9031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:27.297303 9031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:27.301213 9031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:27.301262 9031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:27.301297 9031 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:27.301327 9031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:27.306376 9031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:27.307385 9031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:27.308359 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:27.317783 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:27.317726 9039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:27.317728 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:27.317823 9031 server_base.cc:1034] running on GCE node
I20260926 12:28:27.318351 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:27.318935 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:27.320084 9031 hybrid_clock.cc:648] HybridClock initialized: now 1790425707320056 us; error 40 us; skew 500 ppm
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:27.323161 9031 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:27.325321 9031 webserver.cc:460] Webserver started at http://127.7.99.3:35441/ using document root <none> and password file <none>
I20260926 12:28:27.325866 9031 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:27.325927 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:27.326179 9031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:27.328272 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "839d93f756e5406fbeda8563680d6e43"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "cafec697153149d1ff8b438026216a39"
server_key_iv: "d07fcfe17e0979ec91178a2b23983199"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.328765 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "839d93f756e5406fbeda8563680d6e43"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "cafec697153149d1ff8b438026216a39"
server_key_iv: "d07fcfe17e0979ec91178a2b23983199"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.332240 9031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:28:27.334425 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:27.335604 9031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:28:27.335731 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "839d93f756e5406fbeda8563680d6e43"
format_stamp: "Formatted at 2026-09-26 12:28:27 on dist-test-slave-3jw2"
server_key: "cafec697153149d1ff8b438026216a39"
server_key_iv: "d07fcfe17e0979ec91178a2b23983199"
server_key_version: "encryptionkey@0"
I20260926 12:28:27.335837 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:27.354454 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:27.357299 9031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:27.357499 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:27.359000 9031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:27.359073 9031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:27.359203 9031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:27.359256 9031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:27.371120 9031 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:40751
I20260926 12:28:27.371182 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:40751 every 8 connection(s)
I20260926 12:28:27.372031 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:27.373731 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.359943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:38879 (local address 127.7.99.62:40929)
0926 12:28:27.360065 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.360069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.361171 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 12:28:27.362220 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.362224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.362285 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.362369 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.364373 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.365342 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.366634 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.366919 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.367040 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.369935 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.369939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.369947 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.369974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.371603 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.372107 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.372111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.372113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.372250 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.372621 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.372625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.372628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.372861 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.372969 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.373507 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.373611 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:27.374505 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.360537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.3:38879)
0926 12:28:27.361020 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.361061 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.362004 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.362553 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.362561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.362993 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:27.364155 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.364213 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.365486 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.365489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.366494 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.366504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.367097 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.367101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.367275 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.367925 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:27.367947 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:27.369761 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:27.371716 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.371722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.371737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.371987 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.372339 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.372343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.372345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.372506 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.372970 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:27.372975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:27.373261 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.373425 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.373674 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":142,"threads_started":1}
I20260926 12:28:27.375473 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9031
I20260926 12:28:27.375595 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:27.375700 9162 heartbeater.cc:347] Connected to a master server at 127.7.99.62:40929
I20260926 12:28:27.375829 9162 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:27.375909 7564 external_mini_cluster.cc:1428] Setting key e0d4ecbd3f1b63fbd5a169aa0c0b4013
I20260926 12:28:27.376399 9162 heartbeater.cc:511] Master 127.7.99.62:40929 requested a full tablet report, sending...
I20260926 12:28:27.377470 8679 ts_manager.cc:194] Registered new tserver with Master: 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751)
I20260926 12:28:27.378252 8679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:38879
I20260926 12:28:27.384660 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.378034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.7.99.3:44537)
0926 12:28:27.378347 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.378362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.378455 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.378710 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.378713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.378912 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:27.379006 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.379011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.379996 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.379999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.380628 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.380636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.381019 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.381021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.381084 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.381491 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:27.381501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:27.381807 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:27.383431 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.383433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.383435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.383609 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.383831 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.383833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.383835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.383903 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.384258 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:27.384261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:27.384343 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.384424 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.384521 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260926 12:28:27.384665 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.378105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:44537 (local address 127.7.99.62:40929)
0926 12:28:27.378211 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.378215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.378428 (+ 213us) server_negotiation.cc:407] Connection header received
0926 12:28:27.378539 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.378542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.378604 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.378677 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.379126 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.379871 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.380726 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.380929 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.381006 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.381918 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.381921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.381923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.381942 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.383341 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.383688 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.383691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.383692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.383760 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.383971 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.383974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.383975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.384121 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.384270 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.384462 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.384544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:28:27.389539 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:27.414240 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.406353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.7.99.62:40929)
0926 12:28:27.406468 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.406472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.406484 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:27.406524 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.406527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.406565 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.406642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.407331 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.407910 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.408764 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.408971 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.409042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.411438 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.411441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.411442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.411464 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.412887 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.413241 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.413244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.413245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.413311 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.413516 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.413518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.413519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.413683 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.413779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.413954 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.414036 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 12:28:27.416967 8679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:27.419906 8679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:27.434840 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.428687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32919 (local address 127.0.0.1:44010)
0926 12:28:27.429287 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.429299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.430054 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.430544 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.430547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.430568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.430666 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.430673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431936 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.431939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.432849 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.432858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.434012 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.434016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.434267 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.434354 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.434474 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.434665 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":156,"thread_start_us":83,"threads_started":1}
I20260926 12:28:27.435134 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.428901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.7.99.2:32919)
0926 12:28:27.429294 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.429309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.429478 (+ 169us) server_negotiation.cc:407] Connection header received
0926 12:28:27.430201 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.430208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.430399 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.430543 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.430788 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431808 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.432976 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.433834 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.433948 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.434109 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.434519 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.434627 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":107,"threads_started":1}
I20260926 12:28:27.434840 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.427944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:46787 (local address 127.0.0.1:60784)
0926 12:28:27.428775 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.428826 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.428978 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.429963 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.429966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.429988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.430178 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.430188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431555 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.431558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.432988 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.433000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.434012 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.434016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.434263 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.434354 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.434474 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.434665 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":220,"thread_start_us":82,"threads_started":1}
I20260926 12:28:27.435606 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.428137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.7.99.1:46787)
0926 12:28:27.429544 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.429552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.429574 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:28:27.429646 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.429649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.429812 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.429958 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.430300 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431437 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.433116 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.433834 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.433948 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.434095 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.435344 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.435458 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":94,"threads_started":1}
I20260926 12:28:27.435319 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.428085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40751 (local address 127.0.0.1:34292)
0926 12:28:27.428963 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.428978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.429087 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.429872 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.429881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.429908 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.430078 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.430087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431328 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.431334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.432988 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.433000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.434930 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.434934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.435029 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.435065 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.435115 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.435195 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":108,"thread_start_us":116,"threads_started":1}
I20260926 12:28:27.436337 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.429073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.7.99.3:40751)
0926 12:28:27.429422 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.429435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.429461 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:28:27.429551 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.429555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.429718 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.429851 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.430232 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.431207 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.434157 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.434792 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.435413 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.436012 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.436079 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.436144 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260926 12:28:27.438501 8956 tablet_service.cc:1467] Processing CreateTablet for tablet c3c90d85e575489c8ca1aff861e9667c (DEFAULT_TABLE table=test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.438501 8816 tablet_service.cc:1467] Processing CreateTablet for tablet c3c90d85e575489c8ca1aff861e9667c (DEFAULT_TABLE table=test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.439603 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c90d85e575489c8ca1aff861e9667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.439603 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c90d85e575489c8ca1aff861e9667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.442677 9096 tablet_service.cc:1467] Processing CreateTablet for tablet c3c90d85e575489c8ca1aff861e9667c (DEFAULT_TABLE table=test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.443720 9096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c90d85e575489c8ca1aff861e9667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.446975 9185 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Bootstrap starting.
I20260926 12:28:27.448320 9186 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Bootstrap starting.
I20260926 12:28:27.449290 9185 tablet_bootstrap.cc:654] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.449841 9187 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Bootstrap starting.
I20260926 12:28:27.450129 9185 log.cc:824] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:27.450248 9186 tablet_bootstrap.cc:654] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.451045 9186 log.cc:824] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:27.451689 9187 tablet_bootstrap.cc:654] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.452035 9185 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: No bootstrap required, opened a new log
I20260926 12:28:27.452271 9185 ts_tablet_manager.cc:1397] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:27.452481 9187 log.cc:824] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:27.452987 9186 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: No bootstrap required, opened a new log
I20260926 12:28:27.453208 9186 ts_tablet_manager.cc:1397] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:27.454336 9187 tablet_bootstrap.cc:492] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: No bootstrap required, opened a new log
I20260926 12:28:27.454502 9187 ts_tablet_manager.cc:1397] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:28:27.455626 9185 raft_consensus.cc:348] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.455983 9185 raft_consensus.cc:374] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.456086 9185 raft_consensus.cc:729] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2f150b554f4b9e7c7b9f46330562, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.456621 9185 consensus_queue.cc:260] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.457223 8882 heartbeater.cc:503] Master 127.7.99.62:40929 was elected leader, sending a full tablet report...
I20260926 12:28:27.457623 9185 ts_tablet_manager.cc:1428] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:27.456940 9186 raft_consensus.cc:348] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.458184 9187 raft_consensus.cc:348] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.458380 9186 raft_consensus.cc:374] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.458410 9187 raft_consensus.cc:374] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.458459 9187 raft_consensus.cc:729] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839d93f756e5406fbeda8563680d6e43, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.458477 9186 raft_consensus.cc:729] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebd9fe5929641b6bb4e3b65a563e953, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.458887 9186 consensus_queue.cc:260] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.458887 9187 consensus_queue.cc:260] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.459800 9186 ts_tablet_manager.cc:1428] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:28:27.459800 9187 ts_tablet_manager.cc:1428] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 12:28:27.460242 9162 heartbeater.cc:503] Master 127.7.99.62:40929 was elected leader, sending a full tablet report...
I20260926 12:28:27.460352 9022 heartbeater.cc:503] Master 127.7.99.62:40929 was elected leader, sending a full tablet report...
I20260926 12:28:27.574707 9191 raft_consensus.cc:482] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:27.574921 9191 raft_consensus.cc:504] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.576406 9191 leader_election.cc:288] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751), cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2:32919)
I20260926 12:28:27.580498 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.577059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42349 (local address 127.7.99.2:32919)
0926 12:28:27.577203 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.577209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.577230 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:28:27.577273 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.577275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.577342 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.577413 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.577704 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.578268 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.579430 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.579872 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.579951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.580041 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.580292 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.580366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:28:27.580499 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.576689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:60323 (local address 127.7.99.3:40751)
0926 12:28:27.576841 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.576846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.576996 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:28:27.577199 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.577203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.577261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.577337 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.577667 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.578236 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.579433 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.579888 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.580008 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.580070 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.580311 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.580379 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:27.580531 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.576651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32919 (local address 127.7.99.1:42349)
0926 12:28:27.577090 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.577104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.577197 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.577440 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.577443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.577459 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.577578 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.577583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.578421 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.578423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.579310 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.579318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.579992 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.579996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.580101 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.580123 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.580258 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.580356 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":53,"thread_start_us":121,"threads_started":1}
I20260926 12:28:27.580543 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.576571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40751 (local address 127.7.99.1:60323)
0926 12:28:27.576899 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.576914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.577075 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.577367 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.577373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.577394 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.577553 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.577559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.578362 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.578368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.579308 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.579317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.580030 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.580033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.580146 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.580205 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.580299 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.580391 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":83,"thread_start_us":133,"threads_started":1}
I20260926 12:28:27.581158 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c90d85e575489c8ca1aff861e9667c" candidate_uuid: "f49e2f150b554f4b9e7c7b9f46330562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebd9fe5929641b6bb4e3b65a563e953" is_pre_election: true
I20260926 12:28:27.581310 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c90d85e575489c8ca1aff861e9667c" candidate_uuid: "f49e2f150b554f4b9e7c7b9f46330562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43" is_pre_election: true
I20260926 12:28:27.581449 8976 raft_consensus.cc:2454] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2f150b554f4b9e7c7b9f46330562 in term 0.
I20260926 12:28:27.581585 9116 raft_consensus.cc:2454] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2f150b554f4b9e7c7b9f46330562 in term 0.
I20260926 12:28:27.581909 8768 leader_election.cc:302] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cebd9fe5929641b6bb4e3b65a563e953, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.582211 9191 raft_consensus.cc:2781] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:27.582278 9191 raft_consensus.cc:482] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:27.582306 9191 raft_consensus.cc:3037] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.583316 9191 raft_consensus.cc:504] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.583648 9191 leader_election.cc:288] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [CANDIDATE]: Term 1 election: Requested vote from peers 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751), cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2:32919)
I20260926 12:28:27.583971 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c90d85e575489c8ca1aff861e9667c" candidate_uuid: "f49e2f150b554f4b9e7c7b9f46330562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebd9fe5929641b6bb4e3b65a563e953"
I20260926 12:28:27.583971 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c90d85e575489c8ca1aff861e9667c" candidate_uuid: "f49e2f150b554f4b9e7c7b9f46330562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43"
I20260926 12:28:27.584095 9116 raft_consensus.cc:3037] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.584086 8976 raft_consensus.cc:3037] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.585420 9116 raft_consensus.cc:2454] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2f150b554f4b9e7c7b9f46330562 in term 1.
I20260926 12:28:27.585759 8767 leader_election.cc:302] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 839d93f756e5406fbeda8563680d6e43, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.585897 8976 raft_consensus.cc:2454] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2f150b554f4b9e7c7b9f46330562 in term 1.
I20260926 12:28:27.585942 9191 raft_consensus.cc:2781] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:27.586146 9191 raft_consensus.cc:686] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 1 LEADER]: Becoming Leader. State: Replica: f49e2f150b554f4b9e7c7b9f46330562, State: Running, Role: LEADER
I20260926 12:28:27.586452 9191 consensus_queue.cc:237] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } }
I20260926 12:28:27.589663 8678 catalog_manager.cc:5495] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 reported cstate change: term changed from 0 to 1, leader changed from <none> to f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "f49e2f150b554f4b9e7c7b9f46330562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } health_report { overall_health: UNKNOWN } } }
W20260926 12:28:27.623265 9163 tablet.cc:2369] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:27.625483 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.621656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.7.99.1:46787)
0926 12:28:27.621801 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.621806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.621819 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:28:27.622042 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.622045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.622101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.622204 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:27.622495 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.623020 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.623978 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.624262 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.624371 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.624691 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.625091 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.625212 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.625288 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.625344 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:28:27.630887 8976 raft_consensus.cc:1261] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Refusing update from remote peer f49e2f150b554f4b9e7c7b9f46330562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.630887 9116 raft_consensus.cc:1261] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Refusing update from remote peer f49e2f150b554f4b9e7c7b9f46330562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.631558 9196 consensus_queue.cc:1035] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.631703 9191 consensus_queue.cc:1035] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.639736 9200 mvcc.cc:205] Tried to move back new op lower bound from 7333583698449551360 to 7333583698276696064. Current Snapshot: MvccSnapshot[applied={T|T < 7333583698449551360}]
I20260926 12:28:27.652503 8677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:27.652709 8677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:27.655308 8677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Sending DeleteTablet for 3 replicas of tablet c3c90d85e575489c8ca1aff861e9667c
I20260926 12:28:27.656108 9096 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c90d85e575489c8ca1aff861e9667c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:34292
I20260926 12:28:27.656323 8816 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c90d85e575489c8ca1aff861e9667c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260926 12:28:27.656669 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c90d85e575489c8ca1aff861e9667c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:44010
I20260926 12:28:27.657014 9209 tablet_replica.cc:330] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: stopping tablet replica
I20260926 12:28:27.657094 9211 tablet_replica.cc:330] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: stopping tablet replica
I20260926 12:28:27.657379 9211 raft_consensus.cc:2229] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:27.657437 9210 tablet_replica.cc:330] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: stopping tablet replica
I20260926 12:28:27.657702 9211 raft_consensus.cc:2258] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:27.657757 8677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:28:27.657853 9210 raft_consensus.cc:2229] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 1 LEADER]: Raft consensus shutting down.
W20260926 12:28:27.658172 8677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:27.658502 9210 raft_consensus.cc:2258] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:27.658929 9209 raft_consensus.cc:2229] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:27.659298 9211 ts_tablet_manager.cc:1905] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:27.659497 9209 raft_consensus.cc:2258] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:27.660918 9209 ts_tablet_manager.cc:1905] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:27.662741 9211 ts_tablet_manager.cc:1918] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:27.662842 9211 log.cc:1196] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/c3c90d85e575489c8ca1aff861e9667c
I20260926 12:28:27.663158 9211 ts_tablet_manager.cc:1939] T c3c90d85e575489c8ca1aff861e9667c P cebd9fe5929641b6bb4e3b65a563e953: Deleting consensus metadata
I20260926 12:28:27.663857 9209 ts_tablet_manager.cc:1918] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:27.663825 8816 tablet_service.cc:1467] Processing CreateTablet for tablet ecabb2fbede344c7adcaa945050e44de (DEFAULT_TABLE table=test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.664016 9209 log.cc:1196] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/c3c90d85e575489c8ca1aff861e9667c
I20260926 12:28:27.664069 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecabb2fbede344c7adcaa945050e44de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.664469 9209 ts_tablet_manager.cc:1939] T c3c90d85e575489c8ca1aff861e9667c P 839d93f756e5406fbeda8563680d6e43: Deleting consensus metadata
I20260926 12:28:27.664436 8664 catalog_manager.cc:4841] TS cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2:32919): tablet c3c90d85e575489c8ca1aff861e9667c (table test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]) successfully deleted
I20260926 12:28:27.664860 8956 tablet_service.cc:1467] Processing CreateTablet for tablet ecabb2fbede344c7adcaa945050e44de (DEFAULT_TABLE table=test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.665107 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecabb2fbede344c7adcaa945050e44de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.666456 9185 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Bootstrap starting.
I20260926 12:28:27.667632 9185 tablet_bootstrap.cc:654] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.668030 8663 catalog_manager.cc:4841] TS 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751): tablet c3c90d85e575489c8ca1aff861e9667c (table test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]) successfully deleted
I20260926 12:28:27.668403 9186 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Bootstrap starting.
I20260926 12:28:27.668573 9096 tablet_service.cc:1467] Processing CreateTablet for tablet ecabb2fbede344c7adcaa945050e44de (DEFAULT_TABLE table=test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:27.668807 9096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecabb2fbede344c7adcaa945050e44de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.669435 9186 tablet_bootstrap.cc:654] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.670981 9187 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Bootstrap starting.
I20260926 12:28:27.671936 9187 tablet_bootstrap.cc:654] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.672637 9185 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: No bootstrap required, opened a new log
I20260926 12:28:27.672722 9185 ts_tablet_manager.cc:1397] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 12:28:27.672783 9210 ts_tablet_manager.cc:1905] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:27.673242 9186 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: No bootstrap required, opened a new log
I20260926 12:28:27.673198 9185 raft_consensus.cc:348] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.673308 9186 ts_tablet_manager.cc:1397] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 12:28:27.673306 9185 raft_consensus.cc:374] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.673341 9185 raft_consensus.cc:729] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2f150b554f4b9e7c7b9f46330562, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.673475 9185 consensus_queue.cc:260] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.673713 9185 ts_tablet_manager.cc:1428] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:28:27.673715 9186 raft_consensus.cc:348] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.673811 9186 raft_consensus.cc:374] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.673853 9186 raft_consensus.cc:729] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebd9fe5929641b6bb4e3b65a563e953, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.673995 9186 consensus_queue.cc:260] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.674309 9186 ts_tablet_manager.cc:1428] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:27.674665 9187 tablet_bootstrap.cc:492] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: No bootstrap required, opened a new log
I20260926 12:28:27.674741 9187 ts_tablet_manager.cc:1397] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 12:28:27.675184 9187 raft_consensus.cc:348] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.675307 9187 raft_consensus.cc:374] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.675338 9187 raft_consensus.cc:729] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839d93f756e5406fbeda8563680d6e43, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.675350 9210 ts_tablet_manager.cc:1918] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:27.675451 9210 log.cc:1196] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/c3c90d85e575489c8ca1aff861e9667c
I20260926 12:28:27.675462 9187 consensus_queue.cc:260] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.675736 9187 ts_tablet_manager.cc:1428] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:27.675760 9210 ts_tablet_manager.cc:1939] T c3c90d85e575489c8ca1aff861e9667c P f49e2f150b554f4b9e7c7b9f46330562: Deleting consensus metadata
I20260926 12:28:27.677081 8664 catalog_manager.cc:4841] TS f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787): tablet c3c90d85e575489c8ca1aff861e9667c (table test-table [id=2bf4262574d7423ebd3c0fdc2e6e74e7]) successfully deleted
I20260926 12:28:27.746471 9193 raft_consensus.cc:482] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:27.746639 9193 raft_consensus.cc:504] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.747864 9193 leader_election.cc:288] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787), 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751)
I20260926 12:28:27.752255 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.748292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:52187 (local address 127.7.99.1:46787)
0926 12:28:27.748488 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.748492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.748559 (+ 67us) server_negotiation.cc:407] Connection header received
0926 12:28:27.748757 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.748760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.748810 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.748900 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.749220 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.750040 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.751139 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.751527 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.751653 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.751721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.752030 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.752108 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 12:28:27.752704 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.748096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:46787 (local address 127.7.99.2:52187)
0926 12:28:27.748473 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.748488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.748610 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.748928 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.748931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.748944 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.749086 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.749095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.750190 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.750192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.751000 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.751008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.751780 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.751784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.751894 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.751916 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.752446 (+ 530us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.752524 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":101,"threads_started":1}
I20260926 12:28:27.752916 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.748249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:38803 (local address 127.7.99.3:40751)
0926 12:28:27.748379 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.748384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.748743 (+ 359us) server_negotiation.cc:407] Connection header received
0926 12:28:27.748911 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.748915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.748964 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.749023 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:27.749308 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.750032 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.751617 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.752235 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.752338 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.752394 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.752699 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.752779 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:28:27.753257 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.748096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40751 (local address 127.7.99.2:38803)
0926 12:28:27.748625 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.748638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.748803 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.749065 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.749069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.749088 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:27.749186 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.749192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.750166 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.750170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.751479 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.751489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.752474 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.752477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.752561 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.752582 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.753046 (+ 464us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.753119 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":273,"thread_start_us":64,"threads_started":1}
I20260926 12:28:27.753312 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecabb2fbede344c7adcaa945050e44de" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2f150b554f4b9e7c7b9f46330562" is_pre_election: true
I20260926 12:28:27.753572 8836 raft_consensus.cc:2454] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 0.
I20260926 12:28:27.753741 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecabb2fbede344c7adcaa945050e44de" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43" is_pre_election: true
I20260926 12:28:27.753890 9116 raft_consensus.cc:2454] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 0.
I20260926 12:28:27.754102 8908 leader_election.cc:302] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cebd9fe5929641b6bb4e3b65a563e953, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.754411 9193 raft_consensus.cc:2781] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:27.754473 9193 raft_consensus.cc:482] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:27.754511 9193 raft_consensus.cc:3037] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.755509 9193 raft_consensus.cc:504] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.755915 9193 leader_election.cc:288] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 election: Requested vote from peers f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787), 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751)
I20260926 12:28:27.756325 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecabb2fbede344c7adcaa945050e44de" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43"
I20260926 12:28:27.756347 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecabb2fbede344c7adcaa945050e44de" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2f150b554f4b9e7c7b9f46330562"
I20260926 12:28:27.756465 9116 raft_consensus.cc:3037] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.756479 8836 raft_consensus.cc:3037] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.757479 8836 raft_consensus.cc:2454] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 1.
I20260926 12:28:27.757788 8908 leader_election.cc:302] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cebd9fe5929641b6bb4e3b65a563e953, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.757812 9116 raft_consensus.cc:2454] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 1.
I20260926 12:28:27.757985 9193 raft_consensus.cc:2781] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:27.758194 9193 raft_consensus.cc:686] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 1 LEADER]: Becoming Leader. State: Replica: cebd9fe5929641b6bb4e3b65a563e953, State: Running, Role: LEADER
I20260926 12:28:27.758445 9193 consensus_queue.cc:237] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.760921 8679 catalog_manager.cc:5495] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 reported cstate change: term changed from 0 to 1, leader changed from <none> to cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2). New cstate: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:28:27 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:27.811012 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.801848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40929 (local address 127.0.0.1:49594)
0926 12:28:27.802145 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.802162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.802273 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.802551 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.802553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.802925 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:27.803015 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.803020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.804211 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.804217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.804802 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.804811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.805446 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.805449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.805510 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.806101 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:27.806112 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:27.807667 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:27.809562 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.809570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.809574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.809792 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.810065 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:27.810067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:27.810069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:27.810141 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:27.810545 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:27.810550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:27.810663 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.810757 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.810864 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260926 12:28:27.811136 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.801955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.7.99.62:40929)
0926 12:28:27.802123 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.802129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.802247 (+ 118us) server_negotiation.cc:407] Connection header received
0926 12:28:27.802365 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.802368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.802417 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.802514 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:27.803165 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.804047 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.804960 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.805339 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.805447 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.807811 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:27.807817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:27.807821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:27.807852 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:27.809464 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.809881 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.809884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.809886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.809975 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:27.810245 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:27.810248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:27.810250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:27.810446 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:27.810534 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.810828 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.810936 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:27.815321 8679 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:49594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:28:27.816129 8679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:27.820911 8816 tablet_service.cc:1467] Processing CreateTablet for tablet 05cceb0849a443d781379167067b892f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ca7a809b4f4b4b84ad1cede80dc958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:27.821080 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 05cceb0849a443d781379167067b892f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ca7a809b4f4b4b84ad1cede80dc958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:27.821177 9096 tablet_service.cc:1467] Processing CreateTablet for tablet 05cceb0849a443d781379167067b892f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ca7a809b4f4b4b84ad1cede80dc958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:27.821310 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cceb0849a443d781379167067b892f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.821458 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cceb0849a443d781379167067b892f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.821519 9096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cceb0849a443d781379167067b892f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:27.823734 9186 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: Bootstrap starting.
I20260926 12:28:27.824246 9185 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: Bootstrap starting.
I20260926 12:28:27.824350 9187 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: Bootstrap starting.
I20260926 12:28:27.824728 9186 tablet_bootstrap.cc:654] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.825063 9185 tablet_bootstrap.cc:654] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.825371 9187 tablet_bootstrap.cc:654] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:27.826159 9186 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: No bootstrap required, opened a new log
I20260926 12:28:27.826159 9185 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: No bootstrap required, opened a new log
I20260926 12:28:27.826236 9186 ts_tablet_manager.cc:1397] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 12:28:27.826236 9185 ts_tablet_manager.cc:1397] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:27.826603 9187 tablet_bootstrap.cc:492] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: No bootstrap required, opened a new log
I20260926 12:28:27.826695 9187 ts_tablet_manager.cc:1397] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 12:28:27.826642 9185 raft_consensus.cc:348] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.826653 9186 raft_consensus.cc:348] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.826766 9185 raft_consensus.cc:374] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.826766 9186 raft_consensus.cc:374] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.826803 9186 raft_consensus.cc:729] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebd9fe5929641b6bb4e3b65a563e953, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.826803 9185 raft_consensus.cc:729] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2f150b554f4b9e7c7b9f46330562, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.826928 9186 consensus_queue.cc:260] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.826928 9185 consensus_queue.cc:260] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.827185 9185 ts_tablet_manager.cc:1428] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:27.827185 9186 ts_tablet_manager.cc:1428] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:27.827165 9187 raft_consensus.cc:348] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.827286 9187 raft_consensus.cc:374] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:27.827328 9187 raft_consensus.cc:729] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839d93f756e5406fbeda8563680d6e43, State: Initialized, Role: FOLLOWER
I20260926 12:28:27.827380 9196 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.827457 9187 consensus_queue.cc:260] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.827512 9196 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.827702 9187 ts_tablet_manager.cc:1428] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:27.827706 9192 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.827878 9192 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.827883 9193 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.827997 9193 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 12:28:27.907030 9023 tablet.cc:2369] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:27.913414 9193 raft_consensus.cc:482] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:27.913551 9193 raft_consensus.cc:504] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.914007 9193 leader_election.cc:288] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787), 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751)
I20260926 12:28:27.914459 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cceb0849a443d781379167067b892f" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2f150b554f4b9e7c7b9f46330562" is_pre_election: true
I20260926 12:28:27.914500 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cceb0849a443d781379167067b892f" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43" is_pre_election: true
I20260926 12:28:27.914639 8836 raft_consensus.cc:2454] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 0.
I20260926 12:28:27.914664 9116 raft_consensus.cc:2454] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 0.
I20260926 12:28:27.914921 8908 leader_election.cc:302] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cebd9fe5929641b6bb4e3b65a563e953, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.915099 9193 raft_consensus.cc:2781] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:27.915159 9193 raft_consensus.cc:482] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:27.915215 9193 raft_consensus.cc:3037] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.916198 9193 raft_consensus.cc:504] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.916541 9193 leader_election.cc:288] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 election: Requested vote from peers f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787), 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751)
I20260926 12:28:27.916807 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cceb0849a443d781379167067b892f" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2f150b554f4b9e7c7b9f46330562"
I20260926 12:28:27.916882 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cceb0849a443d781379167067b892f" candidate_uuid: "cebd9fe5929641b6bb4e3b65a563e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839d93f756e5406fbeda8563680d6e43"
I20260926 12:28:27.916937 8836 raft_consensus.cc:3037] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.917011 9116 raft_consensus.cc:3037] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:27.917837 8836 raft_consensus.cc:2454] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 1.
I20260926 12:28:27.917845 9116 raft_consensus.cc:2454] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebd9fe5929641b6bb4e3b65a563e953 in term 1.
I20260926 12:28:27.918114 8908 leader_election.cc:302] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cebd9fe5929641b6bb4e3b65a563e953, f49e2f150b554f4b9e7c7b9f46330562; no voters:
I20260926 12:28:27.918303 9193 raft_consensus.cc:2781] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:27.918413 9193 raft_consensus.cc:686] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [term 1 LEADER]: Becoming Leader. State: Replica: cebd9fe5929641b6bb4e3b65a563e953, State: Running, Role: LEADER
I20260926 12:28:27.918545 9193 consensus_queue.cc:237] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } }
I20260926 12:28:27.919162 9217 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: TxnStatusTablet state changed. Reason: New leader cebd9fe5929641b6bb4e3b65a563e953. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.919279 9217 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:27.920274 8679 catalog_manager.cc:5495] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 reported cstate change: term changed from 0 to 1, leader changed from <none> to cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2). New cstate: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
W20260926 12:28:27.922798 8883 tablet.cc:2369] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:27.924337 9229 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:28:27.924490 9229 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:28:27.955242 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.951214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32919 (local address 127.0.0.1:44020)
0926 12:28:27.951395 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.951409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.951497 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.951835 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.951837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.951845 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:27.951935 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.951939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.952951 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.952954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.953566 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.953574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.954126 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.954128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.954192 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.954235 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.954987 (+ 752us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.954998 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.955048 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.955122 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 12:28:27.955268 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.951313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.7.99.2:32919)
0926 12:28:27.951441 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.951445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.951462 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:28:27.951642 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.951646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.951717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.951822 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:27.952034 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.952836 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.953677 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.954019 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.954128 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.954375 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.954899 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.955009 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.955072 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.955136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:28:27.958444 9116 raft_consensus.cc:1261] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Refusing update from remote peer cebd9fe5929641b6bb4e3b65a563e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.958674 8836 raft_consensus.cc:1261] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Refusing update from remote peer cebd9fe5929641b6bb4e3b65a563e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.958971 9193 consensus_queue.cc:1035] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.959255 9217 consensus_queue.cc:1035] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.961216 9196 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: TxnStatusTablet state changed. Reason: New leader cebd9fe5929641b6bb4e3b65a563e953. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.961352 9196 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.961891 9192 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: TxnStatusTablet state changed. Reason: New leader cebd9fe5929641b6bb4e3b65a563e953. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.962005 9192 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.963147 9193 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.963253 9193 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:27.964610 9192 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.964712 9192 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P 839d93f756e5406fbeda8563680d6e43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.966269 9217 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.966374 9217 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P cebd9fe5929641b6bb4e3b65a563e953: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:27.967264 9196 tablet_replica.cc:439] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cebd9fe5929641b6bb4e3b65a563e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 } } peers { permanent_uuid: "cebd9fe5929641b6bb4e3b65a563e953" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32919 } } peers { permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 } } }
I20260926 12:28:27.967366 9196 tablet_replica.cc:442] T 05cceb0849a443d781379167067b892f P f49e2f150b554f4b9e7c7b9f46330562: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:27.972949 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.969767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.7.99.2:32919)
0926 12:28:27.969869 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.969873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.969932 (+ 59us) server_negotiation.cc:407] Connection header received
0926 12:28:27.970062 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.970065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.970119 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.970195 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:27.970388 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.971164 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.971930 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.972291 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.972387 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.972576 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.972651 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.972718 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.972787 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.972832 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:28:27.982689 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.979569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:56415 (local address 127.7.99.2:32919)
0926 12:28:27.979672 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:27.979676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:27.979820 (+ 144us) server_negotiation.cc:407] Connection header received
0926 12:28:27.979971 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:27.979974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:27.980015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:27.980092 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:27.980350 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.980823 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.981674 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.981876 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.981978 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.982266 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.982329 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.982402 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:27.982478 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 12:28:27.982551 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":36}
I20260926 12:28:27.982704 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:27.979486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32919 (local address 127.7.99.2:56415)
0926 12:28:27.979739 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:27.979754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:27.979883 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:27.980100 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:27.980103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:27.980120 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:27.980257 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.980261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.980923 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.980929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.981567 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:27.981575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:27.981976 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:27.981979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:27.982101 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:27.982135 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:27.982405 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:27.982412 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:28:27.982472 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:28:27.982573 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":45,"thread_start_us":93,"threads_started":1}
I20260926 12:28:27.988547 8836 raft_consensus.cc:1261] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Refusing update from remote peer cebd9fe5929641b6bb4e3b65a563e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.988894 9116 raft_consensus.cc:1261] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Refusing update from remote peer cebd9fe5929641b6bb4e3b65a563e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:27.988965 9234 consensus_queue.cc:1035] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49e2f150b554f4b9e7c7b9f46330562" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.989241 9217 consensus_queue.cc:1035] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "839d93f756e5406fbeda8563680d6e43" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:27.993280 9201 mvcc.cc:205] Tried to move back new op lower bound from 7333583699917627392 to 7333583698981535744. Current Snapshot: MvccSnapshot[applied={T|T < 7333583699917627392}]
I20260926 12:28:27.995988 9199 mvcc.cc:205] Tried to move back new op lower bound from 7333583699917627392 to 7333583698981535744. Current Snapshot: MvccSnapshot[applied={T|T < 7333583699917627392}]
I20260926 12:28:28.025961 8678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:28.026199 8678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:28.028369 8678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c84637f274484d2c9bb3364886bfb51c: Sending DeleteTablet for 3 replicas of tablet ecabb2fbede344c7adcaa945050e44de
I20260926 12:28:28.028913 8816 tablet_service.cc:1514] Processing DeleteTablet for tablet ecabb2fbede344c7adcaa945050e44de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:28 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260926 12:28:28.028934 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ecabb2fbede344c7adcaa945050e44de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:28 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:44010
I20260926 12:28:28.029110 9211 tablet_replica.cc:330] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: stopping tablet replica
I20260926 12:28:28.029115 9210 tablet_replica.cc:330] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: stopping tablet replica
I20260926 12:28:28.029222 9211 raft_consensus.cc:2229] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:28.029251 9210 raft_consensus.cc:2229] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:28.029397 9210 raft_consensus.cc:2258] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:28.029513 9211 raft_consensus.cc:2258] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:28.029568 9096 tablet_service.cc:1514] Processing DeleteTablet for tablet ecabb2fbede344c7adcaa945050e44de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:28 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:34292
I20260926 12:28:28.030295 9209 tablet_replica.cc:330] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: stopping tablet replica
I20260926 12:28:28.030355 9210 ts_tablet_manager.cc:1905] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:28.030421 9209 raft_consensus.cc:2229] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:28.030537 9209 raft_consensus.cc:2258] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:28.031342 9209 ts_tablet_manager.cc:1905] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:28.033317 9210 ts_tablet_manager.cc:1918] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:28.033397 9210 log.cc:1196] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/ecabb2fbede344c7adcaa945050e44de
I20260926 12:28:28.033905 9211 ts_tablet_manager.cc:1905] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:28.034155 9209 ts_tablet_manager.cc:1918] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:28.034216 9209 log.cc:1196] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/ecabb2fbede344c7adcaa945050e44de
I20260926 12:28:28.034482 9209 ts_tablet_manager.cc:1939] T ecabb2fbede344c7adcaa945050e44de P 839d93f756e5406fbeda8563680d6e43: Deleting consensus metadata
I20260926 12:28:28.035163 8663 catalog_manager.cc:4841] TS 839d93f756e5406fbeda8563680d6e43 (127.7.99.3:40751): tablet ecabb2fbede344c7adcaa945050e44de (table test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]) successfully deleted
I20260926 12:28:28.036324 9211 ts_tablet_manager.cc:1918] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:28.036399 9211 log.cc:1196] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/ecabb2fbede344c7adcaa945050e44de
I20260926 12:28:28.036639 9211 ts_tablet_manager.cc:1939] T ecabb2fbede344c7adcaa945050e44de P cebd9fe5929641b6bb4e3b65a563e953: Deleting consensus metadata
I20260926 12:28:28.037412 9210 ts_tablet_manager.cc:1939] T ecabb2fbede344c7adcaa945050e44de P f49e2f150b554f4b9e7c7b9f46330562: Deleting consensus metadata
I20260926 12:28:28.038178 8664 catalog_manager.cc:4841] TS cebd9fe5929641b6bb4e3b65a563e953 (127.7.99.2:32919): tablet ecabb2fbede344c7adcaa945050e44de (table test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]) successfully deleted
I20260926 12:28:28.038337 8664 catalog_manager.cc:4841] TS f49e2f150b554f4b9e7c7b9f46330562 (127.7.99.1:46787): tablet ecabb2fbede344c7adcaa945050e44de (table test-table [id=7e22a52a0a7e4ba4b39211725d73c8a9]) successfully deleted
Sep 26 12:28:28 dist-test-slave-3jw2 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:28.046037 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:28.030523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.7.99.1:46787)
0926 12:28:28.030721 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:28.030725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:28.032446 (+ 1721us) server_negotiation.cc:407] Connection header received
0926 12:28:28.032634 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:28.032638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:28.032686 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:28.032736 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:28.033811 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:28.034588 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:28.036273 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:28.036671 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:28.036790 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:28.040505 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:28.040514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:28.040525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:28.040554 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:28.043774 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:28.044230 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:28.044240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:28.044242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:28.044344 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:28.044754 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:28.044758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:28.044760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:28.045129 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:28.045362 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:28.045725 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 12:28:28.045863 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260926 12:28:28.046464 8876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60802
I20260926 12:28:28.055308 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:28.048206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.7.99.62:40929)
0926 12:28:28.048322 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:28.048327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:28.048419 (+ 92us) server_negotiation.cc:407] Connection header received
0926 12:28:28.048573 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:28.048576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:28.048632 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:28.048716 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:28.049331 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:28.049861 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:28.050651 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:28.050884 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:28.050991 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:28.052350 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:28.052354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:28.052357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:28.052389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:28.053716 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:28.054116 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:28.054120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:28.054122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:28.054208 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:28.054469 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:28.054473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:28.054475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:28.054675 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:28.054801 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:28.055019 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 12:28:28.055134 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:28:28.055691 8678 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49606
I20260926 12:28:28.056931 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8751
I20260926 12:28:28.063867 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8891
I20260926 12:28:28.070719 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9031
I20260926 12:28:28.075963 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 8647
2026-09-26T12:28:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2755 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:28 dist-test-slave-3jw2 krb5kdc[9248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:28 dist-test-slave-3jw2 krb5kdc[9248](info): set up 2 sockets
Sep 26 12:28:28 dist-test-slave-3jw2 krb5kdc[9248](info): commencing operation
krb5kdc: starting...
W20260926 12:28:29.112352 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:29.230612 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45353
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:29.338167 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:29.338400 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:29.338455 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:29.342283 9261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:29.342336 9261 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:28:29.342355 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:29.342372 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:29.342388 9261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:29.342402 9261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:29.347506 9261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:45353
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:29.348495 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:29.349462 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:29.358043 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.358088 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.358088 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:29.358372 9261 server_base.cc:1034] running on GCE node
I20260926 12:28:29.358898 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:29.359977 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:29.361166 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790425709361151 us; error 38 us; skew 500 ppm
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:29.363935 9261 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:29.364854 9261 webserver.cc:460] Webserver started at http://127.7.99.62:35375/ using document root <none> and password file <none>
I20260926 12:28:29.365401 9261 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:29.365491 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:29.365725 9261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:29.367660 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "ea2cfb185ad9422a8037257641c413b6"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "4893d3a7d79d70e21958229eec63802d"
server_key_iv: "0a07899ae48589cb31ef7a3ddc9c7f4a"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.368289 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "ea2cfb185ad9422a8037257641c413b6"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "4893d3a7d79d70e21958229eec63802d"
server_key_iv: "0a07899ae48589cb31ef7a3ddc9c7f4a"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.371858 9261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:28:29.374207 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:29.375394 9261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:29.375535 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "ea2cfb185ad9422a8037257641c413b6"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "4893d3a7d79d70e21958229eec63802d"
server_key_iv: "0a07899ae48589cb31ef7a3ddc9c7f4a"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.375658 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:29.392365 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:29.395504 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:29.395704 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:29.403348 9261 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:45353
I20260926 12:28:29.403367 9338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:45353 every 8 connection(s)
I20260926 12:28:29.404289 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:29.406510 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9261
I20260926 12:28:29.406629 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:29.406929 7564 external_mini_cluster.cc:1428] Setting key 62b9f98dfdb75ac8337208b4c649aa07
I20260926 12:28:29.407199 9339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:29.413229 9339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Bootstrap starting.
I20260926 12:28:29.415565 9339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Neither blocks nor log segments found. Creating new log.
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:29.416371 9339 log.cc:824] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:29.418092 9339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: No bootstrap required, opened a new log
I20260926 12:28:29.421077 9339 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } }
I20260926 12:28:29.421271 9339 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:29.421339 9339 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2cfb185ad9422a8037257641c413b6, State: Initialized, Role: FOLLOWER
I20260926 12:28:29.421341 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.408359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.7.99.62:45353)
0926 12:28:29.408769 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:29.408783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:29.408816 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:28:29.409822 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:29.409840 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:29.410173 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:29.410502 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:29.411293 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.412350 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.413217 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.413716 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.413887 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.416402 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:29.416408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:29.416419 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:29.416444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:29.418651 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.419107 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.419113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.419127 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.419238 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.419524 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.419527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.419529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.419889 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:29.420097 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:29.420365 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:28:29.420627 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260926 12:28:29.421813 9339 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } }
I20260926 12:28:29.421967 9339 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:29.422050 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:29.422173 9339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:29.422992 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } }
I20260926 12:28:29.423343 9339 leader_election.cc:302] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea2cfb185ad9422a8037257641c413b6; no voters:
I20260926 12:28:29.423681 9339 leader_election.cc:288] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:29.423796 9344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:29.424014 9344 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [term 1 LEADER]: Becoming Leader. State: Replica: ea2cfb185ad9422a8037257641c413b6, State: Running, Role: LEADER
I20260926 12:28:29.424364 9344 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } }
I20260926 12:28:29.424896 9339 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:29.425897 9346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea2cfb185ad9422a8037257641c413b6. Latest consensus state: current_term: 1 leader_uuid: "ea2cfb185ad9422a8037257641c413b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } } }
I20260926 12:28:29.426033 9346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:29.426374 9345 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea2cfb185ad9422a8037257641c413b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2cfb185ad9422a8037257641c413b6" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45353 } } }
I20260926 12:28:29.426532 9345 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:29.427153 9353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:29.429872 9353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:29.435065 9353 catalog_manager.cc:1347] Generated new cluster ID: 2e5c677ebf134c158f862c6f39020eb0
I20260926 12:28:29.435144 9353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:29.451334 9353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:29.452566 9353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:29.466709 9353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Generated new TSK 0
I20260926 12:28:29.467535 9353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:29.497558 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45353
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:29.607484 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:29.607762 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:29.607858 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:29.611877 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:29.611963 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:29.612033 9365 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:28:29.612099 9365 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:29.612182 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:29.617570 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:29.618623 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:29.619633 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:29.629006 9370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.628997 9373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.628996 9371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:29.629191 9365 server_base.cc:1034] running on GCE node
I20260926 12:28:29.629590 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:29.630230 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:29.631410 9365 hybrid_clock.cc:648] HybridClock initialized: now 1790425709631393 us; error 27 us; skew 500 ppm
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:29.634033 9365 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:29.634845 9365 webserver.cc:460] Webserver started at http://127.7.99.1:36131/ using document root <none> and password file <none>
I20260926 12:28:29.635382 9365 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:29.635465 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:29.635690 9365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:29.637737 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "1dac27a004ad4bc98c100437a6f65452"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "796c3228041049caf0b14d150879b617"
server_key_iv: "44824e6fbf30524c2f8b4e842e611d79"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.638257 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "1dac27a004ad4bc98c100437a6f65452"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "796c3228041049caf0b14d150879b617"
server_key_iv: "44824e6fbf30524c2f8b4e842e611d79"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.641674 9365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:28:29.644064 9380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:29.645247 9365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:29.645373 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "1dac27a004ad4bc98c100437a6f65452"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "796c3228041049caf0b14d150879b617"
server_key_iv: "44824e6fbf30524c2f8b4e842e611d79"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.645478 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:29.668156 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:29.671356 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:29.671558 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:29.673112 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:29.673182 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:29.673250 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:29.673301 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:29.684062 9365 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:43197
I20260926 12:28:29.684149 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:43197 every 8 connection(s)
I20260926 12:28:29.684978 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:29.686599 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.673816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51343 (local address 127.7.99.62:45353)
0926 12:28:29.673966 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:29.673971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:29.675212 (+ 1241us) server_negotiation.cc:407] Connection header received
0926 12:28:29.676301 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:29.676304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:29.676364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:29.676465 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:29.677652 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.678197 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.679513 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.679835 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.679962 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.682860 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:29.682864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:29.682868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:29.682898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:29.684579 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.685171 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.685174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.685175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.685251 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.685617 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.685622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.685623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.685854 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:29.685971 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:29.686369 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 12:28:29.686484 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:29.687354 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.674623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.1:51343)
0926 12:28:29.675054 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:29.675103 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:29.676035 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:29.676637 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:29.676644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:29.676969 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:29.677511 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.677524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.678322 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.678332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.679380 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.679389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.680005 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.680009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.680235 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.680862 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:29.680881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:29.682697 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:29.684720 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.684727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.684741 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.685028 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.685362 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.685367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.685369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.685519 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.685955 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:29.685959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:29.686163 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 12:28:29.686316 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 12:28:29.686566 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260926 12:28:29.688783 9496 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45353
I20260926 12:28:29.688941 9496 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:29.689431 9496 heartbeater.cc:511] Master 127.7.99.62:45353 requested a full tablet report, sending...
I20260926 12:28:29.690968 9292 ts_manager.cc:194] Registered new tserver with Master: 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197)
I20260926 12:28:29.692342 9292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:51343
I20260926 12:28:29.693815 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9365
I20260926 12:28:29.693933 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:29.694226 7564 external_mini_cluster.cc:1428] Setting key 534618022e3a63e0da9b673f22539c3d
I20260926 12:28:29.697944 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.690783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.1:37451)
0926 12:28:29.691088 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:29.691103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:29.691196 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:29.691409 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:29.691412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:29.691604 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:29.691715 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.691722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.692573 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.692576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.693286 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.693295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.693727 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.693730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.693831 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.694327 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:29.694341 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:29.694701 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:29.696434 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.696440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.696445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.696724 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.696975 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.696981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.696986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.697094 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.697514 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:29.697516 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:29.697595 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 12:28:29.697674 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:28:29.697816 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
I20260926 12:28:29.698055 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.690920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:37451 (local address 127.7.99.62:45353)
0926 12:28:29.691038 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:29.691041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:29.691160 (+ 119us) server_negotiation.cc:407] Connection header received
0926 12:28:29.691274 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:29.691276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:29.691312 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:29.691373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:29.691826 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.692432 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.693417 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.693620 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.693725 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.694848 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:29.694851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:29.694854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:29.694878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:29.696330 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.696820 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.696822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.696824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.696889 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.697180 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.697182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.697183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.697417 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:29.697520 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:29.697752 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 12:28:29.697874 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:29.719245 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45353
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:29.826207 9505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:29.826517 9505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:29.826599 9505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:29.830513 9505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:29.830595 9505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:29.830698 9505 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:28:29.830742 9505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:29.830804 9505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:29.835831 9505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:29.836876 9505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:29.837905 9505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:29.846994 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.846990 9510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:29.846987 9511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:29.847366 9505 server_base.cc:1034] running on GCE node
I20260926 12:28:29.847751 9505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:29.848459 9505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:29.849606 9505 hybrid_clock.cc:648] HybridClock initialized: now 1790425709849575 us; error 41 us; skew 500 ppm
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:29.852329 9505 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:29.853075 9505 webserver.cc:460] Webserver started at http://127.7.99.2:35139/ using document root <none> and password file <none>
I20260926 12:28:29.853567 9505 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:29.853624 9505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:29.853777 9505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:29.855643 9505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "d743e7bd7cda4e65982577369b6ea532"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "8236baa2aa68ed546639522647976a58"
server_key_iv: "83b00f84f2c0a366cf1c49dd834a2378"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.856132 9505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "d743e7bd7cda4e65982577369b6ea532"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "8236baa2aa68ed546639522647976a58"
server_key_iv: "83b00f84f2c0a366cf1c49dd834a2378"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.859548 9505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:28:29.861799 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:29.862933 9505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:29.863049 9505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "d743e7bd7cda4e65982577369b6ea532"
format_stamp: "Formatted at 2026-09-26 12:28:29 on dist-test-slave-3jw2"
server_key: "8236baa2aa68ed546639522647976a58"
server_key_iv: "83b00f84f2c0a366cf1c49dd834a2378"
server_key_version: "encryptionkey@0"
I20260926 12:28:29.863152 9505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:29.875679 9505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:29.878890 9505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:29.879117 9505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:29.880688 9505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:29.880757 9505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:29.880824 9505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:29.880879 9505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:29 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:29.891345 9505 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:40219
I20260926 12:28:29.891404 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:40219 every 8 connection(s)
I20260926 12:28:29.892479 9505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:29.895027 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9505
I20260926 12:28:29.895159 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:29.895285 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.881365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:56897 (local address 127.7.99.62:45353)
0926 12:28:29.881487 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:29.881490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:29.883136 (+ 1646us) server_negotiation.cc:407] Connection header received
0926 12:28:29.884432 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:29.884436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:29.884493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:29.884608 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:29.885806 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.886343 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.887543 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.887766 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.887880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.891624 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:29.891628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:29.891631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:29.891657 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:29.893331 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.893816 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.893821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.893823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.893909 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.894246 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.894250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.894252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.894442 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:29.894548 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:29.895007 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 12:28:29.895127 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:28:29.895512 7564 external_mini_cluster.cc:1428] Setting key a81c90888042c77e4c13780c6dbd4072
I20260926 12:28:29.896136 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.882456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.2:56897)
0926 12:28:29.882987 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:29.883029 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:29.884201 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:29.884795 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:29.884806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:29.885195 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:29.885664 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.885678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.886477 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.886481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.887421 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.887431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.887928 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.887932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.888127 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.889439 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:29.889461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:29.891420 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:29.893444 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.893449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.893463 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.893711 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.894008 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.894011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.894013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.894151 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.894543 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:29.894550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:29.894786 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 12:28:29.894967 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 12:28:29.895274 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
I20260926 12:28:29.897375 9636 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45353
I20260926 12:28:29.897517 9636 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:29.897991 9636 heartbeater.cc:511] Master 127.7.99.62:45353 requested a full tablet report, sending...
I20260926 12:28:29.898991 9292 ts_manager.cc:194] Registered new tserver with Master: d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219)
I20260926 12:28:29.899649 9292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:56897
I20260926 12:28:29.906715 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.899786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:45491 (local address 127.7.99.62:45353)
0926 12:28:29.899905 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:29.899909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:29.900084 (+ 175us) server_negotiation.cc:407] Connection header received
0926 12:28:29.900260 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:29.900263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:29.900303 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:29.900393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:29.900807 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.901308 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.902089 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.902323 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.902429 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.903626 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:29.903629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:29.903631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:29.903652 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:29.905060 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.905565 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.905568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.905569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.905636 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:29.905941 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:29.905943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:29.905944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:29.906136 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:29.906212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:29.906511 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 12:28:29.906608 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:29.906813 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:29.899758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.2:45491)
0926 12:28:29.900005 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:29.900016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:29.900130 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:29.900407 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:29.900410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:29.900607 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:29.900700 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.900705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.901425 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.901429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.901978 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:29.901985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:29.902430 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:29.902434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:29.902536 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:29.902997 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:29.903021 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:29.903485 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:29.905169 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.905175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.905181 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.905458 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.905728 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:29.905734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:29.905738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:29.905843 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:29.906232 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:29.906235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:29.906350 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 12:28:29.906466 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:28:29.906606 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:29.926376 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45353
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:30.035202 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:30.035458 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:30.035503 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:30.039453 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:30.039505 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:30.039543 9645 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:28:30.039563 9645 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:30.039598 9645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:30.044726 9645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:30.045641 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:30.046543 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:30.056526 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:30.056519 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:30.056663 9645 server_base.cc:1034] running on GCE node
W20260926 12:28:30.056559 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:30.057155 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:30.057761 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:30.058913 9645 hybrid_clock.cc:648] HybridClock initialized: now 1790425710058883 us; error 42 us; skew 500 ppm
Sep 26 12:28:30 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:30.061622 9645 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:30.062484 9645 webserver.cc:460] Webserver started at http://127.7.99.3:43523/ using document root <none> and password file <none>
I20260926 12:28:30.063023 9645 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:30.063079 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:30.063306 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:30.065299 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "35972e6721bc4383ab53287a40d6a653"
format_stamp: "Formatted at 2026-09-26 12:28:30 on dist-test-slave-3jw2"
server_key: "481cb9cc2a78b23cb87d91b137526c9a"
server_key_iv: "4d1925b08e6469faa490fcd2eb16fbf0"
server_key_version: "encryptionkey@0"
I20260926 12:28:30.065851 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "35972e6721bc4383ab53287a40d6a653"
format_stamp: "Formatted at 2026-09-26 12:28:30 on dist-test-slave-3jw2"
server_key: "481cb9cc2a78b23cb87d91b137526c9a"
server_key_iv: "4d1925b08e6469faa490fcd2eb16fbf0"
server_key_version: "encryptionkey@0"
I20260926 12:28:30.069365 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:30.071606 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:30.072788 9645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:30.072911 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "35972e6721bc4383ab53287a40d6a653"
format_stamp: "Formatted at 2026-09-26 12:28:30 on dist-test-slave-3jw2"
server_key: "481cb9cc2a78b23cb87d91b137526c9a"
server_key_iv: "4d1925b08e6469faa490fcd2eb16fbf0"
server_key_version: "encryptionkey@0"
I20260926 12:28:30.073014 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:30.087502 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:30.090803 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:30.090992 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:30.092458 9645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:30.092525 9645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:30.092597 9645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:30.092648 9645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:30 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:30.102970 9645 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:40653
I20260926 12:28:30.103153 9775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:40653 every 8 connection(s)
I20260926 12:28:30.103957 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:30.105504 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.093083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:57037 (local address 127.7.99.62:45353)
0926 12:28:30.093239 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.093245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.094468 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 12:28:30.095584 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.095587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.095646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.095725 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:30.097093 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.097625 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.098917 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.099161 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.099275 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.101864 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:30.101870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:30.101874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:30.101904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:30.103417 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.103929 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.103933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.103936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.104015 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.104419 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.104424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.104426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.104669 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:30.104773 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.105285 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.105388 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:30.106194 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.093896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.3:57037)
0926 12:28:30.094313 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.094358 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.095352 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.095893 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.095900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.096256 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:30.096952 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.096975 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.097777 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.097781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.098789 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.098798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.099318 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.099322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.099491 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.100047 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:30.100062 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:30.101701 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:30.103532 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.103541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.103557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.103825 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.104115 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.104118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.104120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.104323 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.104829 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:30.104833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:30.105038 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.105179 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.105405 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260926 12:28:30.107376 9776 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45353
I20260926 12:28:30.107527 9776 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:30.108034 9776 heartbeater.cc:511] Master 127.7.99.62:45353 requested a full tablet report, sending...
I20260926 12:28:30.109007 9293 ts_manager.cc:194] Registered new tserver with Master: 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:30.109731 9293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:57037
I20260926 12:28:30.112300 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9645
I20260926 12:28:30.112432 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:30.112751 7564 external_mini_cluster.cc:1428] Setting key 623693e6005298169257bb9b1d7846b0
I20260926 12:28:30.114593 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:28:30.116139 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.109459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:54895 (local address 127.7.99.62:45353)
0926 12:28:30.109588 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.109592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.109704 (+ 112us) server_negotiation.cc:407] Connection header received
0926 12:28:30.109848 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.109851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.109901 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.109990 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:30.110395 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.110901 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.111716 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.111929 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.112034 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.113176 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:30.113179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:30.113182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:30.113212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:30.114593 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.115007 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.115012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.115014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.115092 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.115360 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.115365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.115367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.115536 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:30.115642 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.115902 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.116004 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:28:30.116151 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.109389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.7.99.3:54895)
0926 12:28:30.109638 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.109652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.109745 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.110004 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.110007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.110198 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:30.110288 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.110293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.111019 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.111049 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.111602 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.111609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.112045 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.112048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.112137 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.112685 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:30.112695 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:30.113044 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:30.114716 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.114719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.114722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.114911 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.115183 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.115187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.115189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.115273 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.115630 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:30.115633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:30.115725 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.115841 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.115971 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
Sep 26 12:28:30 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:28:30 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:30.139964 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.132335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.7.99.62:45353)
0926 12:28:30.132479 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.132482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.132494 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:30.132529 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.132532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.132569 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.132659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:30.133329 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.133851 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.134689 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.134898 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.134999 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.137215 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:30.137225 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:30.137227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:30.137250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:30.138395 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.138805 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.138810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.138811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.138888 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.139151 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.139154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.139155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.139334 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:30.139473 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.139693 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.139803 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:28:30.142611 9293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:30.145601 9293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:30.159152 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.153387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40653 (local address 127.0.0.1:60944)
0926 12:28:30.153761 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.153809 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.153958 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.155406 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.155409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.155432 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.155592 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.155601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.156716 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.156720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.157847 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.157857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.158600 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.158603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.158706 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.158741 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.158882 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.158998 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":25,"spinlock_wait_cycles":14976,"thread_start_us":66,"threads_started":1}
I20260926 12:28:30.159159 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.153521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.7.99.3:40653)
0926 12:28:30.155035 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.155045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.155065 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:28:30.155129 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.155135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.155287 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.155419 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.155719 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.156569 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.157968 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.158480 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.158600 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.158765 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.158912 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.159000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":117,"threads_started":1}
I20260926 12:28:30.160243 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.153568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.0.0.1:50352)
0926 12:28:30.154042 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.154055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.154151 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.155381 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.155390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.155414 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.155650 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.155659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.157127 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.157131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.158611 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.158622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.159681 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.159685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.159924 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.159947 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.160018 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.160098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":219,"spinlock_wait_cycles":1792,"thread_start_us":85,"threads_started":1}
I20260926 12:28:30.160624 9798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.154449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.7.99.1:43197)
0926 12:28:30.154777 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.154787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.154806 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:28:30.154866 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.154870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.155062 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.155215 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.155777 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.156991 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.158751 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.159559 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.159668 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.159846 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.160262 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.160465 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260926 12:28:30.161152 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.154080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40219 (local address 127.0.0.1:53432)
0926 12:28:30.154691 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.154700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.154800 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.155403 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.155406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.155442 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.155629 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.155636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.157127 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.157131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.158611 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.158622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.160784 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.160788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.160879 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.160899 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.160972 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.161033 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":193,"spinlock_wait_cycles":13824,"thread_start_us":77,"threads_started":1}
I20260926 12:28:30.161238 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.154265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.7.99.2:40219)
0926 12:28:30.154610 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.154622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.154786 (+ 164us) server_negotiation.cc:407] Connection header received
0926 12:28:30.154909 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.154914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.155120 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.155258 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.155805 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.156991 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.158751 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.159556 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.159648 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.159796 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.161016 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.161108 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260926 12:28:30.162500 9705 tablet_service.cc:1467] Processing CreateTablet for tablet 055890313adb4c129a157a9f67e48a8f (DEFAULT_TABLE table=test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.163432 9705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055890313adb4c129a157a9f67e48a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.163295 9430 tablet_service.cc:1467] Processing CreateTablet for tablet 055890313adb4c129a157a9f67e48a8f (DEFAULT_TABLE table=test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.163812 9570 tablet_service.cc:1467] Processing CreateTablet for tablet 055890313adb4c129a157a9f67e48a8f (DEFAULT_TABLE table=test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.164342 9430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055890313adb4c129a157a9f67e48a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.164757 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055890313adb4c129a157a9f67e48a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.170260 9799 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Bootstrap starting.
I20260926 12:28:30.171732 9800 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Bootstrap starting.
I20260926 12:28:30.172653 9801 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Bootstrap starting.
I20260926 12:28:30.172840 9799 tablet_bootstrap.cc:654] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.173699 9799 log.cc:824] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:30.173851 9800 tablet_bootstrap.cc:654] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.174432 9801 tablet_bootstrap.cc:654] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.174690 9800 log.cc:824] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:30.175227 9801 log.cc:824] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:30.176436 9799 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: No bootstrap required, opened a new log
I20260926 12:28:30.176594 9800 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: No bootstrap required, opened a new log
I20260926 12:28:30.176641 9799 ts_tablet_manager.cc:1397] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:28:30.176745 9800 ts_tablet_manager.cc:1397] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:30.177155 9801 tablet_bootstrap.cc:492] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: No bootstrap required, opened a new log
I20260926 12:28:30.177335 9801 ts_tablet_manager.cc:1397] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:28:30.180537 9801 raft_consensus.cc:348] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.180573 9800 raft_consensus.cc:348] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.180573 9799 raft_consensus.cc:348] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.180785 9801 raft_consensus.cc:374] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.180810 9799 raft_consensus.cc:374] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.180810 9800 raft_consensus.cc:374] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.180835 9801 raft_consensus.cc:729] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.180861 9800 raft_consensus.cc:729] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.180861 9799 raft_consensus.cc:729] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.181308 9801 consensus_queue.cc:260] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.181309 9800 consensus_queue.cc:260] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.181890 9799 consensus_queue.cc:260] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.182281 9800 ts_tablet_manager.cc:1428] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:30.182281 9801 ts_tablet_manager.cc:1428] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:30.182583 9776 heartbeater.cc:503] Master 127.7.99.62:45353 was elected leader, sending a full tablet report...
I20260926 12:28:30.182600 9496 heartbeater.cc:503] Master 127.7.99.62:45353 was elected leader, sending a full tablet report...
I20260926 12:28:30.183506 9636 heartbeater.cc:503] Master 127.7.99.62:45353 was elected leader, sending a full tablet report...
I20260926 12:28:30.183640 9799 ts_tablet_manager.cc:1428] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 12:28:30.186266 9497 tablet.cc:2369] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:28:30.355404 9777 tablet.cc:2369] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:28:30.393649 9637 tablet.cc:2369] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:30.617666 9805 raft_consensus.cc:482] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:30.617871 9805 raft_consensus.cc:504] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.618981 9805 leader_election.cc:288] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653), d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219)
I20260926 12:28:30.622725 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.619298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:45149 (local address 127.7.99.3:40653)
0926 12:28:30.619429 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.619434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.619516 (+ 82us) server_negotiation.cc:407] Connection header received
0926 12:28:30.619766 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.619770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.619829 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.619884 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.620205 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.620726 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.621777 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.622167 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.622246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.622328 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.622514 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.622596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:28:30.622754 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.619290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40219 (local address 127.7.99.1:56655)
0926 12:28:30.619557 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.619578 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.619679 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.619984 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.619987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.620005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.620103 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.620108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.620858 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.620861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.621661 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.621669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.622324 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.622327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.622424 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.622440 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.622535 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.622595 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260926 12:28:30.622746 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.619474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:56655 (local address 127.7.99.2:40219)
0926 12:28:30.619590 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.619594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.619628 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:28:30.619814 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.619816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.619859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.619953 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.620249 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.620732 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.621810 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.622190 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.622307 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.622366 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.622562 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.622640 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 12:28:30.622761 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.619174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40653 (local address 127.7.99.1:45149)
0926 12:28:30.619430 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.619446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.619611 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.619920 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.619925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.619945 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.620075 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.620082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.620829 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.620832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.621654 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.621662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.622282 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.622286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.622379 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.622407 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.622471 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.622553 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260926 12:28:30.623483 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055890313adb4c129a157a9f67e48a8f" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532" is_pre_election: true
I20260926 12:28:30.623562 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055890313adb4c129a157a9f67e48a8f" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653" is_pre_election: true
I20260926 12:28:30.623814 9730 raft_consensus.cc:2454] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:30.623814 9590 raft_consensus.cc:2454] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:30.624349 9381 leader_election.cc:302] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, 35972e6721bc4383ab53287a40d6a653, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:30.624629 9805 raft_consensus.cc:2781] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:30.624722 9805 raft_consensus.cc:482] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:30.624782 9805 raft_consensus.cc:3037] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.625859 9805 raft_consensus.cc:504] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.626264 9805 leader_election.cc:288] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Requested vote from peers 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653), d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219)
I20260926 12:28:30.626579 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055890313adb4c129a157a9f67e48a8f" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653"
I20260926 12:28:30.626729 9730 raft_consensus.cc:3037] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.627765 9730 raft_consensus.cc:2454] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:30.628069 9381 leader_election.cc:302] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, 35972e6721bc4383ab53287a40d6a653; no voters:
I20260926 12:28:30.628047 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055890313adb4c129a157a9f67e48a8f" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532"
I20260926 12:28:30.628191 9590 raft_consensus.cc:3037] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.628249 9805 raft_consensus.cc:2781] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:30.628440 9805 raft_consensus.cc:686] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 1 LEADER]: Becoming Leader. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Running, Role: LEADER
I20260926 12:28:30.628752 9805 consensus_queue.cc:237] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } }
I20260926 12:28:30.629329 9590 raft_consensus.cc:2454] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:30.631968 9292 catalog_manager.cc:5495] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:28:30 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:30.745362 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.737773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45353 (local address 127.0.0.1:50454)
0926 12:28:30.738038 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.738055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.738158 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.738401 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.738403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.738777 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:30.738877 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.738883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.739604 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.739606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.740149 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.740155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.740615 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.740618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.740688 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.741247 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:30.741258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:30.742604 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:30.744021 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.744031 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.744035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.744320 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.744559 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:30.744561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:30.744562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:30.744636 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:30.744917 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:30.744922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:30.745014 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.745103 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.745212 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260926 12:28:30.745393 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.737855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.7.99.62:45353)
0926 12:28:30.738092 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.738099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.738113 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:28:30.738245 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.738248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.738288 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.738372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:30.738978 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.739497 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.740296 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.740514 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.740600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.742720 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:30.742723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:30.742725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:30.742744 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:30.743929 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.744400 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.744403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.744404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.744475 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:30.744690 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:30.744692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:30.744693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:30.744836 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:30.744905 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.745147 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.745266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
I20260926 12:28:30.748961 9292 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:28:30.749718 9292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:30.754334 9570 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63b133c28fff4d039531ccd8addce0bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:30.754312 9705 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63b133c28fff4d039531ccd8addce0bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:30.754258 9430 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63b133c28fff4d039531ccd8addce0bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:30.754945 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.754943 9430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.754942 9705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d74c6b5024ef79ff8a4049f0f0bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.757586 9819 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: Bootstrap starting.
I20260926 12:28:30.757961 9820 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: Bootstrap starting.
I20260926 12:28:30.758129 9821 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: Bootstrap starting.
I20260926 12:28:30.758769 9819 tablet_bootstrap.cc:654] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.758885 9820 tablet_bootstrap.cc:654] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.759290 9821 tablet_bootstrap.cc:654] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.759989 9819 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: No bootstrap required, opened a new log
I20260926 12:28:30.760097 9819 ts_tablet_manager.cc:1397] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:28:30.760281 9820 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: No bootstrap required, opened a new log
I20260926 12:28:30.760373 9820 ts_tablet_manager.cc:1397] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:30.760546 9821 tablet_bootstrap.cc:492] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: No bootstrap required, opened a new log
I20260926 12:28:30.760624 9821 ts_tablet_manager.cc:1397] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:28:30.760607 9819 raft_consensus.cc:348] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.760720 9819 raft_consensus.cc:374] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.760761 9819 raft_consensus.cc:729] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.760844 9820 raft_consensus.cc:348] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.760878 9819 consensus_queue.cc:260] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.760952 9820 raft_consensus.cc:374] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.760990 9820 raft_consensus.cc:729] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.761075 9821 raft_consensus.cc:348] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.761183 9821 raft_consensus.cc:374] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.761219 9819 ts_tablet_manager.cc:1428] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 12:28:30.761225 9821 raft_consensus.cc:729] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.761155 9820 consensus_queue.cc:260] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.761372 9821 consensus_queue.cc:260] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.761562 9826 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.761611 9805 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.761734 9805 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.761817 9826 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.762236 9825 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.762363 9825 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.762553 9821 ts_tablet_manager.cc:1428] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 12:28:30.763403 9820 ts_tablet_manager.cc:1428] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:30.823771 9805 raft_consensus.cc:482] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:30.823911 9805 raft_consensus.cc:504] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.824325 9805 leader_election.cc:288] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:30.824703 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d74c6b5024ef79ff8a4049f0f0bc8" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532" is_pre_election: true
I20260926 12:28:30.824800 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d74c6b5024ef79ff8a4049f0f0bc8" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653" is_pre_election: true
I20260926 12:28:30.824859 9590 raft_consensus.cc:2454] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:30.824978 9730 raft_consensus.cc:2454] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:30.825198 9384 leader_election.cc:302] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:30.825377 9805 raft_consensus.cc:2781] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:30.825469 9805 raft_consensus.cc:482] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:30.825515 9805 raft_consensus.cc:3037] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.826584 9805 raft_consensus.cc:504] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.826941 9805 leader_election.cc:288] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Requested vote from peers d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:30.827273 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d74c6b5024ef79ff8a4049f0f0bc8" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653"
I20260926 12:28:30.827270 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d74c6b5024ef79ff8a4049f0f0bc8" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532"
I20260926 12:28:30.827363 9730 raft_consensus.cc:3037] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.827399 9590 raft_consensus.cc:3037] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.828316 9730 raft_consensus.cc:2454] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:30.828384 9590 raft_consensus.cc:2454] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:30.828595 9381 leader_election.cc:302] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, 35972e6721bc4383ab53287a40d6a653; no voters:
I20260926 12:28:30.828747 9805 raft_consensus.cc:2781] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:30.828833 9805 raft_consensus.cc:686] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [term 1 LEADER]: Becoming Leader. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Running, Role: LEADER
I20260926 12:28:30.828961 9805 consensus_queue.cc:237] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.829612 9810 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: TxnStatusTablet state changed. Reason: New leader 1dac27a004ad4bc98c100437a6f65452. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.829720 9810 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:30.830037 9828 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:28:30.830276 9828 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:28:30.830607 9292 catalog_manager.cc:5495] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:30.843295 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.839669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.7.99.1:43197)
0926 12:28:30.839959 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.839970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.839982 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:30.840040 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.840042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.840110 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.840260 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:30.840513 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.841047 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.841861 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.842107 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.842212 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.842504 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:30.842894 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:30.843012 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.843087 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.843137 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260926 12:28:30.843278 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.839589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.0.0.1:50368)
0926 12:28:30.839711 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.839726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.839865 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.840282 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.840285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.840296 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:30.840402 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.840407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.841161 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.841166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.841749 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.841756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.842211 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.842214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.842303 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.842360 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:30.842989 (+ 629us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:30.843004 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.843073 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.843158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 12:28:30.848237 9730 raft_consensus.cc:1261] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:30.848235 9590 raft_consensus.cc:1261] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:30.848907 9810 consensus_queue.cc:1035] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:30.849069 9805 consensus_queue.cc:1035] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:30.852459 9826 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: TxnStatusTablet state changed. Reason: New leader 1dac27a004ad4bc98c100437a6f65452. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.852568 9826 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.852669 9825 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: TxnStatusTablet state changed. Reason: New leader 1dac27a004ad4bc98c100437a6f65452. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.852772 9825 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.857573 9805 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.857685 9805 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:30.857980 9825 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.858083 9825 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P d743e7bd7cda4e65982577369b6ea532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.859180 9810 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.859283 9810 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 1dac27a004ad4bc98c100437a6f65452: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:28:30.860302 9832 mvcc.cc:205] Tried to move back new op lower bound from 7333583711627567104 to 7333583711556771840. Current Snapshot: MvccSnapshot[applied={T|T < 7333583711627567104 or (T in {7333583711627567104})}]
I20260926 12:28:30.860385 9826 tablet_replica.cc:439] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } } }
I20260926 12:28:30.860486 9826 tablet_replica.cc:442] T 9e8d74c6b5024ef79ff8a4049f0f0bc8 P 35972e6721bc4383ab53287a40d6a653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:30.860460 9833 mvcc.cc:205] Tried to move back new op lower bound from 7333583711627567104 to 7333583711556771840. Current Snapshot: MvccSnapshot[applied={T|T < 7333583711627567104}]
I20260926 12:28:30.862061 9834 mvcc.cc:205] Tried to move back new op lower bound from 7333583711627567104 to 7333583711556771840. Current Snapshot: MvccSnapshot[applied={T|T < 7333583711627567104}]
I20260926 12:28:30.867702 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.864191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.7.99.1:43197)
0926 12:28:30.864323 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.864327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.864532 (+ 205us) server_negotiation.cc:407] Connection header received
0926 12:28:30.864642 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.864644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.864682 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.864741 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:30.864987 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.865462 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.866701 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.866909 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.866983 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.867232 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:30.867291 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:30.867337 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.867460 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.867518 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:28:30.878072 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.874708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:56669 (local address 127.7.99.1:43197)
0926 12:28:30.874812 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.874818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.874973 (+ 155us) server_negotiation.cc:407] Connection header received
0926 12:28:30.875117 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.875122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.875158 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.875236 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:30.875490 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.876041 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.876927 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.877136 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.877239 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.877552 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:30.877624 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:30.877719 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.877821 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.877926 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":34}
I20260926 12:28:30.878090 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.874648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.7.99.1:56669)
0926 12:28:30.874901 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.874912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.875019 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.875255 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.875258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.875264 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:30.875398 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.875403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.876138 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.876145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.876801 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.876809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.877236 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.877239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.877364 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.877407 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:30.877731 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:30.877739 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.877812 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.877949 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":77,"thread_start_us":103,"threads_started":1}
I20260926 12:28:30.884805 9590 raft_consensus.cc:1261] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:30.884835 9730 raft_consensus.cc:1261] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:30.885151 9805 consensus_queue.cc:1035] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:30.885279 9810 consensus_queue.cc:1035] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:30.934767 9291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:30.934927 9291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:30.937307 9291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Sending DeleteTablet for 3 replicas of tablet 055890313adb4c129a157a9f67e48a8f
I20260926 12:28:30.938009 9705 tablet_service.cc:1514] Processing DeleteTablet for tablet 055890313adb4c129a157a9f67e48a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60944
I20260926 12:28:30.938261 9429 tablet_service.cc:1514] Processing DeleteTablet for tablet 055890313adb4c129a157a9f67e48a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260926 12:28:30.938269 9570 tablet_service.cc:1514] Processing DeleteTablet for tablet 055890313adb4c129a157a9f67e48a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 12:28:30.938588 9849 tablet_replica.cc:330] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: stopping tablet replica
I20260926 12:28:30.938779 9850 tablet_replica.cc:330] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: stopping tablet replica
I20260926 12:28:30.938899 9849 raft_consensus.cc:2229] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:30.939041 9850 raft_consensus.cc:2229] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:30.939106 9849 raft_consensus.cc:2258] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:30.939378 9850 raft_consensus.cc:2258] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:30.939687 9851 tablet_replica.cc:330] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: stopping tablet replica
I20260926 12:28:30.939670 9291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:28:30.939963 9851 raft_consensus.cc:2229] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 12:28:30.940104 9291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:30.940151 9851 raft_consensus.cc:2258] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:30.941094 9849 ts_tablet_manager.cc:1905] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:30.941331 9851 ts_tablet_manager.cc:1905] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:30.944259 9851 ts_tablet_manager.cc:1918] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:30.944259 9849 ts_tablet_manager.cc:1918] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:30.944356 9851 log.cc:1196] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/055890313adb4c129a157a9f67e48a8f
I20260926 12:28:30.944471 9849 log.cc:1196] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/055890313adb4c129a157a9f67e48a8f
I20260926 12:28:30.944670 9851 ts_tablet_manager.cc:1939] T 055890313adb4c129a157a9f67e48a8f P d743e7bd7cda4e65982577369b6ea532: Deleting consensus metadata
I20260926 12:28:30.944943 9849 ts_tablet_manager.cc:1939] T 055890313adb4c129a157a9f67e48a8f P 35972e6721bc4383ab53287a40d6a653: Deleting consensus metadata
I20260926 12:28:30.946110 9570 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 (DEFAULT_TABLE table=test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.946254 9429 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 (DEFAULT_TABLE table=test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.946344 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0cff27f6634615a8fbf52448c204a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.946491 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0cff27f6634615a8fbf52448c204a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.946754 9850 ts_tablet_manager.cc:1905] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:30.948500 9819 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Bootstrap starting.
I20260926 12:28:30.949007 9821 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Bootstrap starting.
I20260926 12:28:30.949178 9705 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 (DEFAULT_TABLE table=test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:30.949437 9819 tablet_bootstrap.cc:654] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.949429 9705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0cff27f6634615a8fbf52448c204a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:30.949851 9821 tablet_bootstrap.cc:654] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.950904 9821 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: No bootstrap required, opened a new log
I20260926 12:28:30.950982 9821 ts_tablet_manager.cc:1397] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 12:28:30.951021 9850 ts_tablet_manager.cc:1918] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:30.951102 9850 log.cc:1196] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/055890313adb4c129a157a9f67e48a8f
I20260926 12:28:30.951511 9820 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Bootstrap starting.
I20260926 12:28:30.951436 9821 raft_consensus.cc:348] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.951560 9821 raft_consensus.cc:374] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.951612 9821 raft_consensus.cc:729] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.951707 9277 catalog_manager.cc:4841] TS 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653): tablet 055890313adb4c129a157a9f67e48a8f (table test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]) successfully deleted
I20260926 12:28:30.951740 9821 consensus_queue.cc:260] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.952013 9821 ts_tablet_manager.cc:1428] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:30.952193 9280 catalog_manager.cc:4841] TS d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219): tablet 055890313adb4c129a157a9f67e48a8f (table test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]) successfully deleted
I20260926 12:28:30.952425 9820 tablet_bootstrap.cc:654] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:30.952968 9850 ts_tablet_manager.cc:1939] T 055890313adb4c129a157a9f67e48a8f P 1dac27a004ad4bc98c100437a6f65452: Deleting consensus metadata
I20260926 12:28:30.954041 9280 catalog_manager.cc:4841] TS 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197): tablet 055890313adb4c129a157a9f67e48a8f (table test-table [id=bc1d989796be419cbf1bf6ecf707ab5b]) successfully deleted
I20260926 12:28:30.954708 9820 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: No bootstrap required, opened a new log
I20260926 12:28:30.954806 9820 ts_tablet_manager.cc:1397] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:30.955210 9820 raft_consensus.cc:348] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.955295 9820 raft_consensus.cc:374] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.955317 9820 raft_consensus.cc:729] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.955439 9820 consensus_queue.cc:260] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.955721 9820 ts_tablet_manager.cc:1428] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:28:30.960915 9819 tablet_bootstrap.cc:492] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: No bootstrap required, opened a new log
I20260926 12:28:30.960994 9819 ts_tablet_manager.cc:1397] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260926 12:28:30.961403 9819 raft_consensus.cc:348] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.961496 9819 raft_consensus.cc:374] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:30.961517 9819 raft_consensus.cc:729] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Initialized, Role: FOLLOWER
I20260926 12:28:30.961671 9819 consensus_queue.cc:260] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.961946 9819 ts_tablet_manager.cc:1428] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:28:30.990252 9825 raft_consensus.cc:482] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:30.990404 9825 raft_consensus.cc:504] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.991482 9825 leader_election.cc:288] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:30.995460 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.991758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:60541 (local address 127.7.99.1:43197)
0926 12:28:30.991885 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.991889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.992067 (+ 178us) server_negotiation.cc:407] Connection header received
0926 12:28:30.992425 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.992430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.992484 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.992532 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.992966 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.993446 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.994490 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.994914 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.995033 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.995110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.995274 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.995346 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:30.995486 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.991645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.7.99.2:60541)
0926 12:28:30.991982 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.991998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.992301 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.992687 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.992689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.992707 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.992839 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.992847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.993555 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.993557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.994365 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.994372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.995032 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.995036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.995142 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.995171 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.995259 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.995330 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"spinlock_wait_cycles":313600,"thread_start_us":114,"threads_started":1}
I20260926 12:28:30.996261 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0cff27f6634615a8fbf52448c204a0" candidate_uuid: "d743e7bd7cda4e65982577369b6ea532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac27a004ad4bc98c100437a6f65452" is_pre_election: true
I20260926 12:28:30.996443 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.991717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40653 (local address 127.7.99.2:38389)
0926 12:28:30.992073 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:30.992085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:30.992206 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:30.992545 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:30.992548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:30.992567 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:30.992691 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.992696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.993741 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.993744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.994945 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:30.994955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.995994 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.995998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:30.996099 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.996126 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:28:30.996250 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 12:28:30.996314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260926 12:28:30.996439 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:30.992112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:38389 (local address 127.7.99.3:40653)
0926 12:28:30.992242 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:30.992247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:30.992263 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:28:30.992314 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:30.992317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:30.992361 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:30.992437 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:30.992811 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.993581 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.995086 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:30.995715 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:30.995842 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:30.995924 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:30.996269 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 12:28:30.996338 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:28:30.996490 9450 raft_consensus.cc:2454] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d743e7bd7cda4e65982577369b6ea532 in term 0.
I20260926 12:28:30.996939 9524 leader_election.cc:302] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:30.997056 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0cff27f6634615a8fbf52448c204a0" candidate_uuid: "d743e7bd7cda4e65982577369b6ea532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653" is_pre_election: true
I20260926 12:28:30.997188 9730 raft_consensus.cc:2454] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d743e7bd7cda4e65982577369b6ea532 in term 0.
I20260926 12:28:30.997215 9825 raft_consensus.cc:2781] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:30.997262 9825 raft_consensus.cc:482] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:30.997303 9825 raft_consensus.cc:3037] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.998747 9825 raft_consensus.cc:504] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:30.999109 9825 leader_election.cc:288] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [CANDIDATE]: Term 1 election: Requested vote from peers 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:30.999418 9730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0cff27f6634615a8fbf52448c204a0" candidate_uuid: "d743e7bd7cda4e65982577369b6ea532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653"
I20260926 12:28:30.999401 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0cff27f6634615a8fbf52448c204a0" candidate_uuid: "d743e7bd7cda4e65982577369b6ea532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac27a004ad4bc98c100437a6f65452"
I20260926 12:28:30.999538 9730 raft_consensus.cc:3037] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:30.999554 9450 raft_consensus.cc:3037] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:31.000455 9730 raft_consensus.cc:2454] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d743e7bd7cda4e65982577369b6ea532 in term 1.
I20260926 12:28:31.000739 9521 leader_election.cc:302] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35972e6721bc4383ab53287a40d6a653, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:31.000741 9450 raft_consensus.cc:2454] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d743e7bd7cda4e65982577369b6ea532 in term 1.
I20260926 12:28:31.000905 9825 raft_consensus.cc:2781] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:31.001116 9825 raft_consensus.cc:686] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 1 LEADER]: Becoming Leader. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Running, Role: LEADER
I20260926 12:28:31.001359 9825 consensus_queue.cc:237] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:31.003726 9292 catalog_manager.cc:5495] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d743e7bd7cda4e65982577369b6ea532 (127.7.99.2). New cstate: current_term: 1 leader_uuid: "d743e7bd7cda4e65982577369b6ea532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:31.028527 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:31.024684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.7.99.2:40219)
0926 12:28:31.024815 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:31.024819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:31.024909 (+ 90us) server_negotiation.cc:407] Connection header received
0926 12:28:31.025086 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:31.025088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:31.025141 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:31.025228 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:31.025459 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.025941 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.026845 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.027227 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.027332 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:31.027544 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:31.028120 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:31.028271 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:31.028332 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:28:31.028387 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:31.039160 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:31.035748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.7.99.2:34199)
0926 12:28:31.036143 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:31.036155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:31.036295 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:31.036544 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:31.036547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:31.036555 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:31.036666 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:31.036674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.037377 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.037380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:31.037987 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:31.037994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.038435 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.038438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:31.038535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:31.038583 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:31.038869 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:31.038877 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 12:28:31.038942 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:28:31.039015 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260926 12:28:31.039137 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:31.035848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:34199 (local address 127.7.99.1:43197)
0926 12:28:31.035987 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:31.035994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:31.036251 (+ 257us) server_negotiation.cc:407] Connection header received
0926 12:28:31.036405 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:31.036408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:31.036449 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:31.036524 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:31.036781 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.037266 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.038120 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.038330 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.038434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:31.038721 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:31.038787 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:31.038865 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:31.038952 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 12:28:31.038997 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:28:31.045159 9730 raft_consensus.cc:1261] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Refusing update from remote peer d743e7bd7cda4e65982577369b6ea532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:31.045336 9450 raft_consensus.cc:1261] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Refusing update from remote peer d743e7bd7cda4e65982577369b6ea532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:31.045899 9857 consensus_queue.cc:1035] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:31.046133 9825 consensus_queue.cc:1035] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:31.067405 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:31.063721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40219 (local address 127.7.99.1:53211)
0926 12:28:31.063953 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:31.063969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:31.064044 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:31.064487 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:31.064491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:31.064498 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:31.064605 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:31.064611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.065281 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.065284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:31.066204 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:31.066214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.066673 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.066677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:31.066759 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:31.066800 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:31.067137 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:31.067146 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:28:31.067204 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 12:28:31.067276 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 12:28:31.067420 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:31.064196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:53211 (local address 127.7.99.2:40219)
0926 12:28:31.064301 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:31.064304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:31.064315 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:28:31.064349 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:31.064351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:31.064391 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:31.064450 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:31.064695 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.065191 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.066326 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:31.066566 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:31.066649 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:31.066961 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:31.067040 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:31.067118 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:31.067223 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 12:28:31.067274 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 12:28:31.068104 9833 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:28:32.626139 9292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:32.626310 9292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:32.628571 9292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Sending DeleteTablet for 3 replicas of tablet 5c0cff27f6634615a8fbf52448c204a0
I20260926 12:28:32.629113 9570 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 12:28:32.629165 9705 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60944
I20260926 12:28:32.629160 9429 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c0cff27f6634615a8fbf52448c204a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260926 12:28:32.629977 9889 tablet_replica.cc:330] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: stopping tablet replica
I20260926 12:28:32.630155 9889 raft_consensus.cc:2229] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:32.630277 9889 raft_consensus.cc:2258] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.630527 9888 tablet_replica.cc:330] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: stopping tablet replica
I20260926 12:28:32.630699 9888 raft_consensus.cc:2229] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:32.630818 9888 raft_consensus.cc:2258] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.630928 9889 ts_tablet_manager.cc:1905] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:32.631058 9887 tablet_replica.cc:330] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: stopping tablet replica
I20260926 12:28:32.631166 9887 raft_consensus.cc:2229] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:32.631480 9887 raft_consensus.cc:2258] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.631531 9888 ts_tablet_manager.cc:1905] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:32.632134 9887 ts_tablet_manager.cc:1905] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 12:28:32 dist-test-slave-3jw2 krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425712, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 12:28:32.673166 9889 ts_tablet_manager.cc:1918] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:28:32.673166 9888 ts_tablet_manager.cc:1918] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:28:32.673265 9889 log.cc:1196] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/5c0cff27f6634615a8fbf52448c204a0
I20260926 12:28:32.673265 9888 log.cc:1196] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/5c0cff27f6634615a8fbf52448c204a0
I20260926 12:28:32.673497 9887 ts_tablet_manager.cc:1918] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:28:32.673534 9888 ts_tablet_manager.cc:1939] T 5c0cff27f6634615a8fbf52448c204a0 P 35972e6721bc4383ab53287a40d6a653: Deleting consensus metadata
I20260926 12:28:32.673534 9889 ts_tablet_manager.cc:1939] T 5c0cff27f6634615a8fbf52448c204a0 P 1dac27a004ad4bc98c100437a6f65452: Deleting consensus metadata
I20260926 12:28:32.673585 9887 log.cc:1196] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/5c0cff27f6634615a8fbf52448c204a0
I20260926 12:28:32.673878 9887 ts_tablet_manager.cc:1939] T 5c0cff27f6634615a8fbf52448c204a0 P d743e7bd7cda4e65982577369b6ea532: Deleting consensus metadata
I20260926 12:28:32.686509 9280 catalog_manager.cc:4841] TS 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197): tablet 5c0cff27f6634615a8fbf52448c204a0 (table test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]) successfully deleted
I20260926 12:28:32.686509 9277 catalog_manager.cc:4841] TS 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653): tablet 5c0cff27f6634615a8fbf52448c204a0 (table test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]) successfully deleted
Sep 26 12:28:32 dist-test-slave-3jw2 krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425712, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:32.691346 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:32.675692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.7.99.62:45353)
0926 12:28:32.675976 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:32.675988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:32.676076 (+ 88us) server_negotiation.cc:407] Connection header received
0926 12:28:32.676281 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:32.676284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:32.676336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:32.676443 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:32.677094 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:32.677721 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:32.678651 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:32.678874 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:32.679003 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:32.687873 (+ 8870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:32.687877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:32.687879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:32.687905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:32.689469 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:32.690098 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:32.690101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:32.690103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:32.690179 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:32.690463 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:32.690468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:32.690470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:32.690672 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:32.690814 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:32.691051 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 12:28:32.691173 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260926 12:28:32.694818 9292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:32.695268 9292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:32.699990 9429 tablet_service.cc:1467] Processing CreateTablet for tablet 8824185652fd439bbb86e6aafe1ea036 (DEFAULT_TABLE table=test-table [id=b79a877b108449c48b9803d905444789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.700054 9570 tablet_service.cc:1467] Processing CreateTablet for tablet 8824185652fd439bbb86e6aafe1ea036 (DEFAULT_TABLE table=test-table [id=b79a877b108449c48b9803d905444789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.700356 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8824185652fd439bbb86e6aafe1ea036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.700363 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8824185652fd439bbb86e6aafe1ea036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.700296 9705 tablet_service.cc:1467] Processing CreateTablet for tablet 8824185652fd439bbb86e6aafe1ea036 (DEFAULT_TABLE table=test-table [id=b79a877b108449c48b9803d905444789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.700567 9705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8824185652fd439bbb86e6aafe1ea036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.702491 9898 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Bootstrap starting.
I20260926 12:28:32.703066 9899 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Bootstrap starting.
I20260926 12:28:32.703342 9898 tablet_bootstrap.cc:654] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.703866 9900 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Bootstrap starting.
I20260926 12:28:32.703955 9899 tablet_bootstrap.cc:654] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.704890 9898 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: No bootstrap required, opened a new log
I20260926 12:28:32.704952 9900 tablet_bootstrap.cc:654] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.705008 9898 ts_tablet_manager.cc:1397] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:32.705053 9899 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: No bootstrap required, opened a new log
I20260926 12:28:32.705147 9899 ts_tablet_manager.cc:1397] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:32.705487 9899 raft_consensus.cc:348] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.705605 9899 raft_consensus.cc:374] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.705538 9898 raft_consensus.cc:348] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.705647 9899 raft_consensus.cc:729] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.705669 9898 raft_consensus.cc:374] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.705714 9898 raft_consensus.cc:729] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.705798 9899 consensus_queue.cc:260] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.705868 9898 consensus_queue.cc:260] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.706194 9898 ts_tablet_manager.cc:1428] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:28:32.706614 9899 ts_tablet_manager.cc:1428] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:32.707393 9900 tablet_bootstrap.cc:492] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: No bootstrap required, opened a new log
I20260926 12:28:32.707580 9900 ts_tablet_manager.cc:1397] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 12:28:32.708142 9900 raft_consensus.cc:348] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.708316 9900 raft_consensus.cc:374] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.708370 9900 raft_consensus.cc:729] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.708556 9900 consensus_queue.cc:260] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.708981 9900 ts_tablet_manager.cc:1428] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:32.719465 9280 catalog_manager.cc:4841] TS d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219): tablet 5c0cff27f6634615a8fbf52448c204a0 (table test-table [id=6332ed28930a4d0ab5ad97c81ce7e500]) successfully deleted
I20260926 12:28:32.822294 9810 raft_consensus.cc:482] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:32.822448 9810 raft_consensus.cc:504] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.822944 9810 leader_election.cc:288] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:32.823405 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8824185652fd439bbb86e6aafe1ea036" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532" is_pre_election: true
I20260926 12:28:32.823482 9729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8824185652fd439bbb86e6aafe1ea036" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653" is_pre_election: true
I20260926 12:28:32.823572 9590 raft_consensus.cc:2454] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:32.823670 9729 raft_consensus.cc:2454] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 0.
I20260926 12:28:32.823967 9384 leader_election.cc:302] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:32.824193 9810 raft_consensus.cc:2781] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:32.824255 9810 raft_consensus.cc:482] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:32.824304 9810 raft_consensus.cc:3037] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:32.869894 9810 raft_consensus.cc:504] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.870539 9810 leader_election.cc:288] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Requested vote from peers d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219), 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653)
I20260926 12:28:32.871001 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8824185652fd439bbb86e6aafe1ea036" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532"
I20260926 12:28:32.871011 9729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8824185652fd439bbb86e6aafe1ea036" candidate_uuid: "1dac27a004ad4bc98c100437a6f65452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35972e6721bc4383ab53287a40d6a653"
I20260926 12:28:32.871150 9729 raft_consensus.cc:3037] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:32.871157 9590 raft_consensus.cc:3037] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:32.872582 9590 raft_consensus.cc:2454] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:32.872767 9729 raft_consensus.cc:2454] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dac27a004ad4bc98c100437a6f65452 in term 1.
I20260926 12:28:32.873001 9384 leader_election.cc:302] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dac27a004ad4bc98c100437a6f65452, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:32.873261 9810 raft_consensus.cc:2781] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:32.873359 9810 raft_consensus.cc:686] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 1 LEADER]: Becoming Leader. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Running, Role: LEADER
I20260926 12:28:32.873507 9810 consensus_queue.cc:237] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.875326 9291 catalog_manager.cc:5495] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "1dac27a004ad4bc98c100437a6f65452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:32.903349 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:32.899896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.7.99.1:43197)
0926 12:28:32.900147 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:32.900154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:32.900194 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:28:32.900237 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:32.900240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:32.900287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:32.900364 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:32.900587 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:32.901396 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:32.902224 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:32.902577 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:32.902675 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:32.902898 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:32.902979 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:32.903049 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:32.903145 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:28:32.903194 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260926 12:28:32.909895 9590 raft_consensus.cc:1261] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:32.909907 9729 raft_consensus.cc:1261] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Refusing update from remote peer 1dac27a004ad4bc98c100437a6f65452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:32.910352 9810 consensus_queue.cc:1035] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:32.910509 9868 consensus_queue.cc:1035] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:32.941223 9292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:32.941365 9292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:32.943136 9292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Sending DeleteTablet for 3 replicas of tablet 8824185652fd439bbb86e6aafe1ea036
I20260926 12:28:32.943625 9429 tablet_service.cc:1514] Processing DeleteTablet for tablet 8824185652fd439bbb86e6aafe1ea036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260926 12:28:32.943704 9570 tablet_service.cc:1514] Processing DeleteTablet for tablet 8824185652fd439bbb86e6aafe1ea036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 12:28:32.943823 9889 tablet_replica.cc:330] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: stopping tablet replica
I20260926 12:28:32.943877 9705 tablet_service.cc:1514] Processing DeleteTablet for tablet 8824185652fd439bbb86e6aafe1ea036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60944
I20260926 12:28:32.943945 9889 raft_consensus.cc:2229] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:32.944079 9887 tablet_replica.cc:330] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: stopping tablet replica
I20260926 12:28:32.944141 9889 raft_consensus.cc:2258] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.944243 9887 raft_consensus.cc:2229] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:32.944360 9887 raft_consensus.cc:2258] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.944522 9888 tablet_replica.cc:330] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: stopping tablet replica
I20260926 12:28:32.944635 9888 raft_consensus.cc:2229] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:32.944434 9292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:28:32.944742 9888 raft_consensus.cc:2258] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:32.944824 9889 ts_tablet_manager.cc:1905] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:32.944973 9887 ts_tablet_manager.cc:1905] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:28:32.944971 9292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:32.945389 9888 ts_tablet_manager.cc:1905] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:32.949183 9429 tablet_service.cc:1467] Processing CreateTablet for tablet e0a26730891448dd8d8e635ab56ea097 (DEFAULT_TABLE table=test-table [id=425096d7a43d473f8eecc867618cb956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.949277 9570 tablet_service.cc:1467] Processing CreateTablet for tablet e0a26730891448dd8d8e635ab56ea097 (DEFAULT_TABLE table=test-table [id=425096d7a43d473f8eecc867618cb956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.949316 9705 tablet_service.cc:1467] Processing CreateTablet for tablet e0a26730891448dd8d8e635ab56ea097 (DEFAULT_TABLE table=test-table [id=425096d7a43d473f8eecc867618cb956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:32.971375 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a26730891448dd8d8e635ab56ea097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.971400 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a26730891448dd8d8e635ab56ea097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.972676 9887 ts_tablet_manager.cc:1918] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:32.972754 9887 log.cc:1196] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/8824185652fd439bbb86e6aafe1ea036
I20260926 12:28:32.973016 9887 ts_tablet_manager.cc:1939] T 8824185652fd439bbb86e6aafe1ea036 P d743e7bd7cda4e65982577369b6ea532: Deleting consensus metadata
I20260926 12:28:32.973629 9705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a26730891448dd8d8e635ab56ea097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:32.974387 9889 ts_tablet_manager.cc:1918] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:32.974453 9889 log.cc:1196] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/8824185652fd439bbb86e6aafe1ea036
I20260926 12:28:32.974710 9889 ts_tablet_manager.cc:1939] T 8824185652fd439bbb86e6aafe1ea036 P 1dac27a004ad4bc98c100437a6f65452: Deleting consensus metadata
I20260926 12:28:32.975035 9898 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Bootstrap starting.
I20260926 12:28:32.975951 9898 tablet_bootstrap.cc:654] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.976032 9899 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452: Bootstrap starting.
I20260926 12:28:32.976220 9900 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Bootstrap starting.
I20260926 12:28:32.976883 9899 tablet_bootstrap.cc:654] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.977144 9900 tablet_bootstrap.cc:654] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:32.977381 9898 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: No bootstrap required, opened a new log
I20260926 12:28:32.977484 9898 ts_tablet_manager.cc:1397] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:32.977919 9899 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452: No bootstrap required, opened a new log
I20260926 12:28:32.978029 9899 ts_tablet_manager.cc:1397] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:32.977957 9898 raft_consensus.cc:348] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.978080 9898 raft_consensus.cc:374] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.978120 9898 raft_consensus.cc:729] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d743e7bd7cda4e65982577369b6ea532, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.978251 9898 consensus_queue.cc:260] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.978418 9900 tablet_bootstrap.cc:492] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: No bootstrap required, opened a new log
I20260926 12:28:32.978509 9900 ts_tablet_manager.cc:1397] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:32.978446 9899 raft_consensus.cc:348] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.978549 9898 ts_tablet_manager.cc:1428] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:32.978552 9899 raft_consensus.cc:374] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.978609 9899 raft_consensus.cc:729] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac27a004ad4bc98c100437a6f65452, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.978736 9899 consensus_queue.cc:260] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.978994 9899 ts_tablet_manager.cc:1428] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:28:32.978933 9900 raft_consensus.cc:348] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.979043 9900 raft_consensus.cc:374] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:32.979097 9900 raft_consensus.cc:729] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Initialized, Role: FOLLOWER
I20260926 12:28:32.979220 9900 consensus_queue.cc:260] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:32.979466 9900 ts_tablet_manager.cc:1428] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:33.020807 9280 catalog_manager.cc:4841] TS d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219): tablet 8824185652fd439bbb86e6aafe1ea036 (table test-table [id=b79a877b108449c48b9803d905444789]) successfully deleted
I20260926 12:28:33.053294 9280 catalog_manager.cc:4841] TS 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197): tablet 8824185652fd439bbb86e6aafe1ea036 (table test-table [id=b79a877b108449c48b9803d905444789]) successfully deleted
I20260926 12:28:33.067669 9904 raft_consensus.cc:482] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:33.067806 9904 raft_consensus.cc:504] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:33.068935 9904 leader_election.cc:288] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197), d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219)
I20260926 12:28:33.072337 9888 ts_tablet_manager.cc:1918] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:28:33.072422 9888 log.cc:1196] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/8824185652fd439bbb86e6aafe1ea036
I20260926 12:28:33.072719 9888 ts_tablet_manager.cc:1939] T 8824185652fd439bbb86e6aafe1ea036 P 35972e6721bc4383ab53287a40d6a653: Deleting consensus metadata
I20260926 12:28:33.073961 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.069381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:59835 (local address 127.7.99.1:43197)
0926 12:28:33.069516 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:33.069523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:33.069565 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:28:33.069772 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:33.069775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:33.069828 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:33.069905 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:33.070260 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.070858 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.072323 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.073102 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.073221 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.073316 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:33.073684 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 12:28:33.073782 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:33.074113 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.069670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:51795 (local address 127.7.99.2:40219)
0926 12:28:33.069905 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:33.069919 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:33.070003 (+ 84us) server_negotiation.cc:407] Connection header received
0926 12:28:33.070101 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:33.070103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:33.070152 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:33.070238 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:33.070499 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.071072 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.073010 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.073475 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.073599 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.073660 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:33.073891 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 12:28:33.073962 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":56,"threads_started":1}
I20260926 12:28:33.074380 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.069591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40219 (local address 127.7.99.3:51795)
0926 12:28:33.069917 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:33.069930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:33.070017 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:33.070258 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:33.070261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:33.070277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:33.070395 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.070401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.071190 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.071192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.072889 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.072898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.073608 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.073611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.073746 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.073774 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:28:33.073862 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:28:33.073930 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":816,"thread_start_us":97,"threads_started":1}
I20260926 12:28:33.074528 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.069181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.7.99.3:59835)
0926 12:28:33.069470 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:33.069486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:33.069658 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:33.069945 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:33.069949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:33.069996 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:33.070144 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.070150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.070956 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.070959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.072179 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.072194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.073377 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.073384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.073523 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.073564 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:28:33.074104 (+ 540us) client_negotiation.cc:240] Negotiation successful
0926 12:28:33.074205 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260926 12:28:33.075081 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a26730891448dd8d8e635ab56ea097" candidate_uuid: "35972e6721bc4383ab53287a40d6a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532" is_pre_election: true
I20260926 12:28:33.075240 9590 raft_consensus.cc:2454] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35972e6721bc4383ab53287a40d6a653 in term 0.
I20260926 12:28:33.075269 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a26730891448dd8d8e635ab56ea097" candidate_uuid: "35972e6721bc4383ab53287a40d6a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac27a004ad4bc98c100437a6f65452" is_pre_election: true
I20260926 12:28:33.075440 9450 raft_consensus.cc:2454] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35972e6721bc4383ab53287a40d6a653 in term 0.
I20260926 12:28:33.075670 9664 leader_election.cc:302] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35972e6721bc4383ab53287a40d6a653, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:33.075961 9904 raft_consensus.cc:2781] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:33.076040 9904 raft_consensus.cc:482] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:33.076102 9904 raft_consensus.cc:3037] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:33.085973 9277 catalog_manager.cc:4841] TS 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653): tablet 8824185652fd439bbb86e6aafe1ea036 (table test-table [id=b79a877b108449c48b9803d905444789]) successfully deleted
I20260926 12:28:33.173010 9904 raft_consensus.cc:504] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:33.173561 9904 leader_election.cc:288] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [CANDIDATE]: Term 1 election: Requested vote from peers 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197), d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219)
I20260926 12:28:33.174146 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a26730891448dd8d8e635ab56ea097" candidate_uuid: "35972e6721bc4383ab53287a40d6a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac27a004ad4bc98c100437a6f65452"
I20260926 12:28:33.174165 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a26730891448dd8d8e635ab56ea097" candidate_uuid: "35972e6721bc4383ab53287a40d6a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d743e7bd7cda4e65982577369b6ea532"
I20260926 12:28:33.174299 9590 raft_consensus.cc:3037] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:33.174299 9450 raft_consensus.cc:3037] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:33.175504 9450 raft_consensus.cc:2454] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35972e6721bc4383ab53287a40d6a653 in term 1.
I20260926 12:28:33.175505 9590 raft_consensus.cc:2454] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35972e6721bc4383ab53287a40d6a653 in term 1.
I20260926 12:28:33.175907 9664 leader_election.cc:302] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35972e6721bc4383ab53287a40d6a653, d743e7bd7cda4e65982577369b6ea532; no voters:
I20260926 12:28:33.176191 9904 raft_consensus.cc:2781] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:33.176421 9904 raft_consensus.cc:686] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 1 LEADER]: Becoming Leader. State: Replica: 35972e6721bc4383ab53287a40d6a653, State: Running, Role: LEADER
I20260926 12:28:33.176716 9904 consensus_queue.cc:237] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } }
I20260926 12:28:33.179741 9293 catalog_manager.cc:5495] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35972e6721bc4383ab53287a40d6a653 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "35972e6721bc4383ab53287a40d6a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35972e6721bc4383ab53287a40d6a653" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 40653 } health_report { overall_health: HEALTHY } } }
I20260926 12:28:33.196357 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.192984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.7.99.3:40653)
0926 12:28:33.193247 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:33.193253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:33.193265 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:33.193331 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:33.193333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:33.193381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:33.193491 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:33.193676 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.194192 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.194921 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.195153 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.195271 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.195465 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:33.195935 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:33.196026 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:33.196073 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 12:28:33.196117 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260926 12:28:33.206079 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.202654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.7.99.3:47453)
0926 12:28:33.202976 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:33.202988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:33.203101 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:33.203355 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:33.203358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:33.203365 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:33.203482 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.203486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.204209 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.204212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.204894 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.204903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.205342 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.205345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.205445 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.205484 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:33.205782 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:33.205788 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:28:33.205843 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 12:28:33.205914 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260926 12:28:33.206056 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.202775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:47453 (local address 127.7.99.1:43197)
0926 12:28:33.202905 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:33.202910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:33.203054 (+ 144us) server_negotiation.cc:407] Connection header received
0926 12:28:33.203204 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:33.203206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:33.203263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:33.203335 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:33.203582 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.204084 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.205013 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.205241 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.205345 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.205626 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:33.205696 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:33.205768 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:33.205863 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 12:28:33.205913 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:28:33.211612 9450 raft_consensus.cc:1261] T e0a26730891448dd8d8e635ab56ea097 P 1dac27a004ad4bc98c100437a6f65452 [term 1 FOLLOWER]: Refusing update from remote peer 35972e6721bc4383ab53287a40d6a653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:33.211612 9590 raft_consensus.cc:1261] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Refusing update from remote peer 35972e6721bc4383ab53287a40d6a653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:33.212100 9904 consensus_queue.cc:1035] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dac27a004ad4bc98c100437a6f65452" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:33.212292 9917 consensus_queue.cc:1035] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d743e7bd7cda4e65982577369b6ea532" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:33.228544 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.225135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:34519 (local address 127.7.99.3:40653)
0926 12:28:33.225413 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:33.225418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:33.225458 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:28:33.225710 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:33.225713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:33.225769 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:33.225844 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:33.226062 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.226650 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.227533 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.227773 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.227880 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.228122 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:33.228227 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:33.228307 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:33.228395 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 12:28:33.228436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260926 12:28:33.228566 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:33.225080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:40653 (local address 127.7.99.1:34519)
0926 12:28:33.225391 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:33.225406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:33.225502 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:33.225863 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:33.225866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:33.225872 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:28:33.225957 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.225961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.226738 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.226741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.227419 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:33.227428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:33.227879 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:33.227883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:33.227965 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:33.227996 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:28:33.228315 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:28:33.228321 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:28:33.228370 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 12:28:33.228431 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
W20260926 12:28:33.229206 9870 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:28:34.792500 9292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:34.792651 9292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:34.794876 9292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea2cfb185ad9422a8037257641c413b6: Sending DeleteTablet for 3 replicas of tablet e0a26730891448dd8d8e635ab56ea097
I20260926 12:28:34.795372 9705 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a26730891448dd8d8e635ab56ea097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:34 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:60944
I20260926 12:28:34.795431 9429 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a26730891448dd8d8e635ab56ea097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:34 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260926 12:28:34.795459 9570 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a26730891448dd8d8e635ab56ea097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:34 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 12:28:34.795773 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9365
I20260926 12:28:34.796635 9950 tablet_replica.cc:330] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: stopping tablet replica
I20260926 12:28:34.796911 9950 raft_consensus.cc:2229] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:34.797154 9952 tablet_replica.cc:330] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: stopping tablet replica
I20260926 12:28:34.797257 9950 raft_consensus.cc:2258] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:34.797339 9952 raft_consensus.cc:2229] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:34.797461 9952 raft_consensus.cc:2258] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:34.798077 9950 ts_tablet_manager.cc:1905] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:34.798084 9952 ts_tablet_manager.cc:1905] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:34.800752 9952 ts_tablet_manager.cc:1918] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:28:34.800752 9950 ts_tablet_manager.cc:1918] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:28:34.800830 9952 log.cc:1196] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/e0a26730891448dd8d8e635ab56ea097
I20260926 12:28:34.800830 9950 log.cc:1196] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/e0a26730891448dd8d8e635ab56ea097
I20260926 12:28:34.801103 9952 ts_tablet_manager.cc:1939] T e0a26730891448dd8d8e635ab56ea097 P d743e7bd7cda4e65982577369b6ea532: Deleting consensus metadata
I20260926 12:28:34.801112 9950 ts_tablet_manager.cc:1939] T e0a26730891448dd8d8e635ab56ea097 P 35972e6721bc4383ab53287a40d6a653: Deleting consensus metadata
I20260926 12:28:34.801716 9280 catalog_manager.cc:4841] TS d743e7bd7cda4e65982577369b6ea532 (127.7.99.2:40219): tablet e0a26730891448dd8d8e635ab56ea097 (table test-table [id=425096d7a43d473f8eecc867618cb956]) successfully deleted
I20260926 12:28:34.801779 9277 catalog_manager.cc:4841] TS 35972e6721bc4383ab53287a40d6a653 (127.7.99.3:40653): tablet e0a26730891448dd8d8e635ab56ea097 (table test-table [id=425096d7a43d473f8eecc867618cb956]) successfully deleted
W20260926 12:28:34.803982 9280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:43197 (error 108)
I20260926 12:28:34.804324 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9505
I20260926 12:28:34.805573 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:34.805094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:43197 (local address 127.0.0.1:50384)
0926 12:28:34.805358 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:34.805445 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
W20260926 12:28:34.805730 9280 catalog_manager.cc:4584] TS 1dac27a004ad4bc98c100437a6f65452 (127.7.99.1:43197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0a26730891448dd8d8e635ab56ea097: Network error: Client connection negotiation failed: client connection to 127.7.99.1:43197: connect: Connection refused (error 111)
I20260926 12:28:34.811309 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9645
I20260926 12:28:34.817965 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9261
2026-09-26T12:28:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6742 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:34 dist-test-slave-3jw2 krb5kdc[9957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:34 dist-test-slave-3jw2 krb5kdc[9957](info): set up 2 sockets
Sep 26 12:28:34 dist-test-slave-3jw2 krb5kdc[9957](info): commencing operation
krb5kdc: starting...
W20260926 12:28:35.852311 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:35 dist-test-slave-3jw2 krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:35.971045 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:35365
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:36.082363 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:36.082649 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:36.082731 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:36.086782 9970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:36.086860 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:36.086898 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:36.086953 9970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:36.086990 9970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:36.092329 9970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:35365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:36.093391 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:36.094450 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:36.103271 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.103329 9970 server_base.cc:1034] running on GCE node
W20260926 12:28:36.103313 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.103267 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.104031 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:36.105194 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:36.106379 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790425716106369 us; error 30 us; skew 500 ppm
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:36.109088 9970 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:36.109916 9970 webserver.cc:460] Webserver started at http://127.7.99.62:36383/ using document root <none> and password file <none>
I20260926 12:28:36.110482 9970 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:36.110587 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:36.110834 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:36.112943 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "9b85843a912f4b81addf7397c8028416"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "56e50882a66dbd52c8728452f6dfbdc3"
server_key_iv: "48d0d9d93b718d0dc2d848eb30551e67"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.113497 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "9b85843a912f4b81addf7397c8028416"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "56e50882a66dbd52c8728452f6dfbdc3"
server_key_iv: "48d0d9d93b718d0dc2d848eb30551e67"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.117265 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:36.119591 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.120865 9970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 12:28:36.121003 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "9b85843a912f4b81addf7397c8028416"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "56e50882a66dbd52c8728452f6dfbdc3"
server_key_iv: "48d0d9d93b718d0dc2d848eb30551e67"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.121114 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:36.144616 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:36.147986 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:36.148269 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:36.155292 9970 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:35365
I20260926 12:28:36.155316 10037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:35365 every 8 connection(s)
I20260926 12:28:36.156271 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:36.156970 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 9970
I20260926 12:28:36.157078 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:36.157351 7564 external_mini_cluster.cc:1428] Setting key 7ccf22a88c479778e258ae78dcf597e9
I20260926 12:28:36.159493 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:36.164988 10038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416: Bootstrap starting.
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:36.167760 10038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:36.168582 10038 log.cc:824] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:36.170526 10038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416: No bootstrap required, opened a new log
I20260926 12:28:36.171499 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.158785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.7.99.62:35365)
0926 12:28:36.159211 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:36.159226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:36.159261 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:28:36.159956 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:36.159973 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:36.160382 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:36.160718 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:36.161495 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.162484 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.163340 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.163805 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.163977 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.166828 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:36.166832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:36.166845 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:36.166872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:36.168694 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.169185 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.169191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.169197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.169308 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.169605 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.169608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.169610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.169996 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:36.170210 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:36.170472 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 12:28:36.170773 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260926 12:28:36.174327 10038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } }
I20260926 12:28:36.174547 10038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:36.174610 10038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b85843a912f4b81addf7397c8028416, State: Initialized, Role: FOLLOWER
I20260926 12:28:36.175015 10038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } }
I20260926 12:28:36.175153 10038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:36.175213 10038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:36.175331 10038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:36.176292 10038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } }
I20260926 12:28:36.176638 10038 leader_election.cc:302] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b85843a912f4b81addf7397c8028416; no voters:
I20260926 12:28:36.176963 10038 leader_election.cc:288] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:36.177115 10043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:36.177347 10043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [term 1 LEADER]: Becoming Leader. State: Replica: 9b85843a912f4b81addf7397c8028416, State: Running, Role: LEADER
I20260926 12:28:36.177683 10043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } }
I20260926 12:28:36.178196 10038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:36.179364 10045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b85843a912f4b81addf7397c8028416. Latest consensus state: current_term: 1 leader_uuid: "9b85843a912f4b81addf7397c8028416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } } }
I20260926 12:28:36.179500 10045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:36.179805 10044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b85843a912f4b81addf7397c8028416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b85843a912f4b81addf7397c8028416" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35365 } } }
I20260926 12:28:36.179905 10044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:36.180142 10052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:36.182739 10052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:36.187862 10052 catalog_manager.cc:1347] Generated new cluster ID: dd87f9ac745c40258b3f978bfaf3027d
I20260926 12:28:36.187927 10052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:36.194509 10052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:36.195341 10052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:36.203660 10052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b85843a912f4b81addf7397c8028416: Generated new TSK 0
I20260926 12:28:36.204371 10052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:36.234766 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:36.342485 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:36.342788 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:36.342860 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:36.346771 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:36.346849 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:36.346937 10064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:36.352073 10064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:36.353080 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:36.354065 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:36.363399 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.363456 10064 server_base.cc:1034] running on GCE node
W20260926 12:28:36.363472 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.363818 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.364255 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:36.364858 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:36.366021 10064 hybrid_clock.cc:648] HybridClock initialized: now 1790425716366010 us; error 22 us; skew 500 ppm
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:36.368510 10064 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:36.369288 10064 webserver.cc:460] Webserver started at http://127.7.99.1:42389/ using document root <none> and password file <none>
I20260926 12:28:36.369776 10064 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:36.369828 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:36.369978 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:36.371910 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "1116ce1d081b43aebec52b99d671019d"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "479c02fbe17c38b7749e712c1b7c504d"
server_key_iv: "f09a0eaa264356bfbc772f280e786e48"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.372457 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "1116ce1d081b43aebec52b99d671019d"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "479c02fbe17c38b7749e712c1b7c504d"
server_key_iv: "f09a0eaa264356bfbc772f280e786e48"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.375797 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:28:36.378188 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.379447 10064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:36.379594 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "1116ce1d081b43aebec52b99d671019d"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "479c02fbe17c38b7749e712c1b7c504d"
server_key_iv: "f09a0eaa264356bfbc772f280e786e48"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.379703 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:36.409157 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:36.412038 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:36.412276 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:36.412894 10064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:36.413705 10064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:36.413748 10064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.413815 10064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:36.413844 10064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.422287 10064 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:35449
I20260926 12:28:36.422312 10192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:35449 every 8 connection(s)
I20260926 12:28:36.423096 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:36.431084 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10064
I20260926 12:28:36.431195 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:36.431473 7564 external_mini_cluster.cc:1428] Setting key 6db628d1cb56129d5eb45b0631567a67
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:36.436779 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.424823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:39139 (local address 127.7.99.62:35365)
0926 12:28:36.424946 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:36.424961 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:36.425782 (+ 821us) server_negotiation.cc:407] Connection header received
0926 12:28:36.427047 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:36.427050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:36.427104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:36.427179 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:36.428283 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.428836 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.430071 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.430303 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.430412 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.433033 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:36.433037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:36.433040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:36.433066 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:36.434682 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.435277 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.435279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.435281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.435352 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.435732 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.435734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.435735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.435887 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:36.435957 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:36.436573 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 12:28:36.436669 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:36.437937 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.425150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35365 (local address 127.7.99.1:39139)
0926 12:28:36.425650 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:36.425690 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:36.426846 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:36.427336 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:36.427342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:36.427662 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:36.428104 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.428116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.428958 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.428961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.429949 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.429959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.430446 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.430450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.430621 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.431214 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:36.431238 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:36.432891 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:36.434807 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.434814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.434829 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.435173 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.435436 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.435441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.435444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.435636 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.435987 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:36.435996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:36.436327 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 12:28:36.436541 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 12:28:36.436867 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260926 12:28:36.439152 10193 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35365
I20260926 12:28:36.439370 10193 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:36.439817 10193 heartbeater.cc:511] Master 127.7.99.62:35365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:28:36.441475 10001 ts_manager.cc:194] Registered new tserver with Master: 1116ce1d081b43aebec52b99d671019d (127.7.99.1:35449)
I20260926 12:28:36.442557 10001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:39139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:36.458680 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:36.580371 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:36.580698 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:36.580780 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:36.586688 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:36.586768 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:36.586869 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:36.592633 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:36.593814 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:36.594883 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:36.603971 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.604010 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.603984 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.604231 10198 server_base.cc:1034] running on GCE node
I20260926 12:28:36.604684 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:36.605332 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:36.606515 10198 hybrid_clock.cc:648] HybridClock initialized: now 1790425716606504 us; error 36 us; skew 500 ppm
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:36.612622 10198 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:36.613797 10198 webserver.cc:460] Webserver started at http://127.7.99.2:33833/ using document root <none> and password file <none>
I20260926 12:28:36.614534 10198 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:36.614619 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:36.614897 10198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:36.617455 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "28e7b64a04c549c6a0dbfb3ceba6cb7e"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "2505ea17b3905b120a9c0effb7818739"
server_key_iv: "d048eb65ca52dfc43a7da833886c77ac"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.618000 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "28e7b64a04c549c6a0dbfb3ceba6cb7e"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "2505ea17b3905b120a9c0effb7818739"
server_key_iv: "d048eb65ca52dfc43a7da833886c77ac"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.621568 10198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:36.623819 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.625165 10198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:36.625290 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "28e7b64a04c549c6a0dbfb3ceba6cb7e"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "2505ea17b3905b120a9c0effb7818739"
server_key_iv: "d048eb65ca52dfc43a7da833886c77ac"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.625396 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:36.649160 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:36.652053 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:36.652307 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:36.652894 10198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:36.653710 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:36.653782 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.653853 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:36.653906 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.662825 10198 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:42313
I20260926 12:28:36.662847 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:42313 every 8 connection(s)
I20260926 12:28:36.663707 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:36.664786 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10198
I20260926 12:28:36.664912 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:36.665258 7564 external_mini_cluster.cc:1428] Setting key 0f2fc03d99ba713820b624d59dabad13
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:28:36.677531 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.665667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:40715 (local address 127.7.99.62:35365)
0926 12:28:36.665831 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:36.665836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:36.666682 (+ 846us) server_negotiation.cc:407] Connection header received
0926 12:28:36.668019 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:36.668023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:36.668087 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:36.668224 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:36.669380 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.669906 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.671388 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.671605 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.671715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.674255 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:36.674259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:36.674261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:36.674284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:36.675494 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.675992 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.675997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.675999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.676079 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.676421 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.676426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.676428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.676663 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:36.676767 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:36.677247 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 12:28:36.677361 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:28:36.678468 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.666087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35365 (local address 127.7.99.2:40715)
0926 12:28:36.666541 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:36.666591 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:36.667803 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:36.668393 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:36.668400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:36.668734 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:36.669231 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.669245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.670030 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.670033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.671253 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.671263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.671750 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.671755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.671933 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.672605 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:36.672620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:36.674084 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:36.675632 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.675637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.675650 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.675890 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.676186 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.676192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.676196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.676328 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.676777 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:36.676786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:36.677013 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 12:28:36.677190 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 12:28:36.677488 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260926 12:28:36.679724 10327 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35365
I20260926 12:28:36.679977 10327 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:36.680655 10327 heartbeater.cc:511] Master 127.7.99.62:35365 requested a full tablet report, sending...
I20260926 12:28:36.681785 10001 ts_manager.cc:194] Registered new tserver with Master: 28e7b64a04c549c6a0dbfb3ceba6cb7e (127.7.99.2:42313)
I20260926 12:28:36.682363 10001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:40715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:36.691409 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:36.803537 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:36.803857 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:36.803946 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:36.808019 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:36.808111 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:36.808234 10332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:36.813454 10332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:36.814575 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:36.815609 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:36.825471 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.825527 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:36.825505 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:36.825618 10332 server_base.cc:1034] running on GCE node
I20260926 12:28:36.826145 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:36.826768 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:36.827950 10332 hybrid_clock.cc:648] HybridClock initialized: now 1790425716827939 us; error 34 us; skew 500 ppm
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:36.830809 10332 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:36.831692 10332 webserver.cc:460] Webserver started at http://127.7.99.3:43819/ using document root <none> and password file <none>
I20260926 12:28:36.832288 10332 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:36.832373 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:36.832581 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:36.834527 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "cf8fa5927cdc4c3d90a885c97bbc5ad1"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "23dfeb725b69b060b3938d1ea9d1bc9e"
server_key_iv: "a5a8f1d6bb54c136164f6147ee69f9ea"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.835068 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "cf8fa5927cdc4c3d90a885c97bbc5ad1"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "23dfeb725b69b060b3938d1ea9d1bc9e"
server_key_iv: "a5a8f1d6bb54c136164f6147ee69f9ea"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.838526 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:28:36.840747 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.841849 10332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:36.841976 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "cf8fa5927cdc4c3d90a885c97bbc5ad1"
format_stamp: "Formatted at 2026-09-26 12:28:36 on dist-test-slave-3jw2"
server_key: "23dfeb725b69b060b3938d1ea9d1bc9e"
server_key_iv: "a5a8f1d6bb54c136164f6147ee69f9ea"
server_key_version: "encryptionkey@0"
I20260926 12:28:36.842078 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:36.863747 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:36.867369 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:36.867596 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:36.868261 10332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:36.869195 10332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:36.869271 10332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.869361 10332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:36.869416 10332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:36.879575 10332 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:35813
I20260926 12:28:36.879596 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:35813 every 8 connection(s)
I20260926 12:28:36.880597 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:36.887419 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10332
I20260926 12:28:36.887545 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:36.887817 7564 external_mini_cluster.cc:1428] Setting key 09f5c15871439a4a99b9a73483fb96b4
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425716, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:36.893748 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.882527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:50531 (local address 127.7.99.62:35365)
0926 12:28:36.882671 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:36.882676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:36.883587 (+ 911us) server_negotiation.cc:407] Connection header received
0926 12:28:36.885005 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:36.885008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:36.885063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:36.885143 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:36.886266 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.886801 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.888070 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.888362 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.888475 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.890881 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:36.890884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:36.890886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:36.890909 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:36.892068 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.892544 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.892547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.892548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.892627 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:36.892896 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:36.892898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:36.892899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:36.893053 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:36.893126 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:36.893539 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 12:28:36.893634 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:36.894532 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:36.882989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35365 (local address 127.7.99.3:50531)
0926 12:28:36.883425 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:36.883465 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:36.884800 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:36.885310 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:36.885316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:36.885661 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:36.886125 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.886137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.886923 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.886927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.887946 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:36.887956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:36.888507 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:36.888511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:36.888681 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:36.889288 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:36.889304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:36.890742 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:36.892200 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.892211 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.892222 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.892453 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.892702 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:36.892704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:36.892706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:36.892824 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:36.893135 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:36.893139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:36.893350 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 12:28:36.893498 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 12:28:36.893725 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260926 12:28:36.895700 10461 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35365
I20260926 12:28:36.895898 10461 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:36.896363 10461 heartbeater.cc:511] Master 127.7.99.62:35365 requested a full tablet report, sending...
I20260926 12:28:36.897286 10001 ts_manager.cc:194] Registered new tserver with Master: cf8fa5927cdc4c3d90a885c97bbc5ad1 (127.7.99.3:35813)
I20260926 12:28:36.897827 10001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:50531
I20260926 12:28:36.901420 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:28:36.913467 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:36.912076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.7.99.62:35365)
0926 12:28:36.912255 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:36.912260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:36.912420 (+ 160us) server_negotiation.cc:407] Connection header received
0926 12:28:36.912531 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:36.912534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:36.912578 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:36.912653 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:36.913309 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44002: BlockingRecv error: recv got EOF from 127.0.0.1:44002 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:36.914609 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10064
I20260926 12:28:36.920615 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10198
I20260926 12:28:36.926004 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10332
I20260926 12:28:36.931710 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 9970
2026-09-26T12:28:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2112 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[10473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[10473](info): set up 2 sockets
Sep 26 12:28:36 dist-test-slave-3jw2 krb5kdc[10473](info): commencing operation
krb5kdc: starting...
W20260926 12:28:37.988576 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:38.110862 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41775
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:38.220584 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:38.220852 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:38.220898 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:38.224953 10486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:38.225010 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:38.225029 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:38.225045 10486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:38.225060 10486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:38.230297 10486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:41775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:38.231285 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:38.232337 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:28:38.240518 10486 server_base.cc:1034] running on GCE node
W20260926 12:28:38.240474 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:38.240494 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:38.240505 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:38.241423 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:38.242403 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:38.243566 10486 hybrid_clock.cc:648] HybridClock initialized: now 1790425718243539 us; error 29 us; skew 500 ppm
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425718, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:38.249544 10486 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:38.250399 10486 webserver.cc:460] Webserver started at http://127.7.99.62:36347/ using document root <none> and password file <none>
I20260926 12:28:38.250954 10486 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:38.251013 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:38.251238 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:38.253284 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "b48cdf1f04a9414ea6c9984d1f0b3774"
format_stamp: "Formatted at 2026-09-26 12:28:38 on dist-test-slave-3jw2"
server_key: "1356376cc934af51b94f65eb24615413"
server_key_iv: "6288d00a1fc847d67d3de32118530a99"
server_key_version: "encryptionkey@0"
I20260926 12:28:38.253834 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "b48cdf1f04a9414ea6c9984d1f0b3774"
format_stamp: "Formatted at 2026-09-26 12:28:38 on dist-test-slave-3jw2"
server_key: "1356376cc934af51b94f65eb24615413"
server_key_iv: "6288d00a1fc847d67d3de32118530a99"
server_key_version: "encryptionkey@0"
I20260926 12:28:38.257349 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:38.259572 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:38.260833 10486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:38.260954 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "b48cdf1f04a9414ea6c9984d1f0b3774"
format_stamp: "Formatted at 2026-09-26 12:28:38 on dist-test-slave-3jw2"
server_key: "1356376cc934af51b94f65eb24615413"
server_key_iv: "6288d00a1fc847d67d3de32118530a99"
server_key_version: "encryptionkey@0"
I20260926 12:28:38.261054 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:38.274113 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:38.277377 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:38.277587 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:38.284385 10486 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:41775
I20260926 12:28:38.284662 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:41775 every 8 connection(s)
I20260926 12:28:38.285423 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:38.286763 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10486
I20260926 12:28:38.286855 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:38.287127 7564 external_mini_cluster.cc:1428] Setting key 397c1d46e31e857b93654fc10e4b7e39
I20260926 12:28:38.288514 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:38.295077 10554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774: Bootstrap starting.
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:38.297582 10554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:38.298274 10554 log.cc:824] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:38.300086 10554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774: No bootstrap required, opened a new log
I20260926 12:28:38.302060 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:38.288435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.7.99.62:41775)
0926 12:28:38.288869 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:38.288888 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:38.288928 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:28:38.289735 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:38.289757 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:38.290167 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:38.290515 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:38.291152 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:38.292640 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:38.293519 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:38.294207 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:38.294371 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:38.297155 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:38.297161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:38.297176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:38.297205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:38.299279 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:38.299734 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:38.299739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:38.299743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:38.299846 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:38.300176 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:38.300183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:38.300184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:38.300595 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:38.300791 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:38.301080 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 12:28:38.301350 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260926 12:28:38.304025 10554 raft_consensus.cc:348] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } }
I20260926 12:28:38.304320 10554 raft_consensus.cc:374] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:38.304389 10554 raft_consensus.cc:729] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48cdf1f04a9414ea6c9984d1f0b3774, State: Initialized, Role: FOLLOWER
I20260926 12:28:38.304792 10554 consensus_queue.cc:260] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } }
I20260926 12:28:38.304986 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:28:38.304986 10554 raft_consensus.cc:388] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:38.305053 10554 raft_consensus.cc:482] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:38.305153 10554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:38.306031 10554 raft_consensus.cc:504] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } }
I20260926 12:28:38.306365 10554 leader_election.cc:302] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b48cdf1f04a9414ea6c9984d1f0b3774; no voters:
I20260926 12:28:38.306718 10554 leader_election.cc:288] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:38.306824 10560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:38.307046 10560 raft_consensus.cc:686] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [term 1 LEADER]: Becoming Leader. State: Replica: b48cdf1f04a9414ea6c9984d1f0b3774, State: Running, Role: LEADER
I20260926 12:28:38.307375 10560 consensus_queue.cc:237] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } }
I20260926 12:28:38.308077 10554 sys_catalog.cc:564] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:38.309077 10561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } } }
I20260926 12:28:38.309080 10562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b48cdf1f04a9414ea6c9984d1f0b3774. Latest consensus state: current_term: 1 leader_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48cdf1f04a9414ea6c9984d1f0b3774" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41775 } } }
I20260926 12:28:38.309223 10562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:38.309223 10561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:38.309703 10569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:38.313004 10569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:38.318439 10569 catalog_manager.cc:1347] Generated new cluster ID: 7f30d03348f8438ca7f54ff4a1661716
I20260926 12:28:38.318572 10569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:38.324555 10569 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:38.325680 10569 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:38.329092 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:38.324237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.7.99.62:41775)
0926 12:28:38.324341 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:38.324345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:38.324638 (+ 293us) server_negotiation.cc:407] Connection header received
0926 12:28:38.324845 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:38.324849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:38.324897 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:38.324985 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:38.325494 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:38.326073 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:38.326907 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:38.327157 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:38.327267 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:38.328111 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:38.328119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:38.328121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:38.328629 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:38.328657 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:38.328674 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:38.328773 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:38.328855 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 12:28:38.328931 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:28:38.333683 10569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b48cdf1f04a9414ea6c9984d1f0b3774: Generated new TSK 0
I20260926 12:28:38.334386 10569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:28:38.343993 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10486
2026-09-26T12:28:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1410 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10587](info): set up 2 sockets
Sep 26 12:28:38 dist-test-slave-3jw2 krb5kdc[10587](info): commencing operation
krb5kdc: starting...
W20260926 12:28:39.380394 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:39 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:39.497166 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45923
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:39.604956 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:39.605288 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:39.605371 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:39.609392 10600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:39.609491 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:39.609536 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:39.609602 10600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:39.609644 10600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:39.614957 10600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:39.616135 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:39.617219 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:39.625628 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:39.625671 10600 server_base.cc:1034] running on GCE node
W20260926 12:28:39.625619 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:39.625619 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:39.626318 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:39.627328 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:39.628481 10600 hybrid_clock.cc:648] HybridClock initialized: now 1790425719628466 us; error 34 us; skew 500 ppm
Sep 26 12:28:39 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425719, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:39.631122 10600 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:39.631886 10600 webserver.cc:460] Webserver started at http://127.7.99.62:44985/ using document root <none> and password file <none>
I20260926 12:28:39.632480 10600 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:39.632546 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:39.632762 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:39.634655 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "4d13b69f4a9644d7a5fedbdd557bda98"
format_stamp: "Formatted at 2026-09-26 12:28:39 on dist-test-slave-3jw2"
server_key: "f17cb3a0ac5f17bebba6775505b988b0"
server_key_iv: "2a96842b7109f33d91ee7a76d96d7096"
server_key_version: "encryptionkey@0"
I20260926 12:28:39.635203 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "4d13b69f4a9644d7a5fedbdd557bda98"
format_stamp: "Formatted at 2026-09-26 12:28:39 on dist-test-slave-3jw2"
server_key: "f17cb3a0ac5f17bebba6775505b988b0"
server_key_iv: "2a96842b7109f33d91ee7a76d96d7096"
server_key_version: "encryptionkey@0"
I20260926 12:28:39.638849 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260926 12:28:39.641059 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:39.642222 10600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:28:39.642341 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "4d13b69f4a9644d7a5fedbdd557bda98"
format_stamp: "Formatted at 2026-09-26 12:28:39 on dist-test-slave-3jw2"
server_key: "f17cb3a0ac5f17bebba6775505b988b0"
server_key_iv: "2a96842b7109f33d91ee7a76d96d7096"
server_key_version: "encryptionkey@0"
I20260926 12:28:39.642436 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:39.656395 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:39.659539 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:39.659683 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:39.666193 10600 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:45923
I20260926 12:28:39.666461 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:45923 every 8 connection(s)
I20260926 12:28:39.667222 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:39.670096 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:39.672791 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10600
I20260926 12:28:39.672920 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:39.673218 7564 external_mini_cluster.cc:1428] Setting key db56998a86753d94918c5d7f2f93a29a
I20260926 12:28:39.675907 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98: Bootstrap starting.
I20260926 12:28:39.678447 10668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:39.679282 10668 log.cc:824] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:39.681304 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98: No bootstrap required, opened a new log
Sep 26 12:28:39 dist-test-slave-3jw2 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:39.684831 10668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } }
I20260926 12:28:39.685030 10668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:39.685097 10668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d13b69f4a9644d7a5fedbdd557bda98, State: Initialized, Role: FOLLOWER
I20260926 12:28:39.685622 10668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } }
I20260926 12:28:39.685753 10668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:39.685835 10668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:39.685945 10668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:39.686882 10668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } }
I20260926 12:28:39.687232 10668 leader_election.cc:302] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d13b69f4a9644d7a5fedbdd557bda98; no voters:
I20260926 12:28:39.687539 10668 leader_election.cc:288] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:39.687636 10673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:39.687860 10673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [term 1 LEADER]: Becoming Leader. State: Replica: 4d13b69f4a9644d7a5fedbdd557bda98, State: Running, Role: LEADER
I20260926 12:28:39.688140 10673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } }
I20260926 12:28:39.688410 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:39.674533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.7.99.62:45923)
0926 12:28:39.674937 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:39.674953 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:39.675010 (+ 57us) server_negotiation.cc:407] Connection header received
0926 12:28:39.675641 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:39.675657 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:39.676001 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:39.676352 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:39.677237 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:39.678181 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:39.679371 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:39.679859 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:39.679981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:39.682795 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:39.682800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:39.682812 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:39.682837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:39.685112 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:39.685598 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:39.685603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:39.685607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:39.685764 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:39.686137 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:39.686139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:39.686140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:39.686517 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:39.686649 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:39.687020 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 12:28:39.687361 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":137,"threads_started":1}
I20260926 12:28:39.689303 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } } }
I20260926 12:28:39.689486 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:39.689738 10668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:39.689853 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d13b69f4a9644d7a5fedbdd557bda98. Latest consensus state: current_term: 1 leader_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d13b69f4a9644d7a5fedbdd557bda98" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45923 } } }
I20260926 12:28:39.690003 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:39.691268 10677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:39.694010 10677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:39.699239 10677 catalog_manager.cc:1347] Generated new cluster ID: 4ac20f45b68d46d1afdefa1cd053433c
I20260926 12:28:39.699301 10677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:39.715494 10677 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:39.716789 10677 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:39.723688 10677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d13b69f4a9644d7a5fedbdd557bda98: Generated new TSK 0
I20260926 12:28:39.724498 10677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
W20260926 12:28:40.563753 7564 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.7.99.1: real 0.832s user 0.000s sys 0.004s
I20260926 12:28:40.564306 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:40.672580 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:40.672849 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:40.672924 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:40.676914 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:40.676997 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:40.677093 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:40.682262 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:40.683250 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:40.684278 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:40.693840 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:40.693869 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:40.693912 10694 server_base.cc:1034] running on GCE node
W20260926 12:28:40.693847 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:40.694489 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:40.695084 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:40.696229 10694 hybrid_clock.cc:648] HybridClock initialized: now 1790425720696198 us; error 44 us; skew 500 ppm
Sep 26 12:28:40 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425720, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:40.698705 10694 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:40.699537 10694 webserver.cc:460] Webserver started at http://127.7.99.1:42061/ using document root <none> and password file <none>
I20260926 12:28:40.700059 10694 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:40.700145 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:40.700377 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:40.702283 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "2203bad9766b428cb371abd406f6f0a0"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "10977cb7cbaa5ce86b78321f605b3d09"
server_key_iv: "43bbdc136f34bbb97c0ebc525b01d3d6"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.702824 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "2203bad9766b428cb371abd406f6f0a0"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "10977cb7cbaa5ce86b78321f605b3d09"
server_key_iv: "43bbdc136f34bbb97c0ebc525b01d3d6"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.706369 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:28:40.709268 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.710714 10694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:40.710834 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "2203bad9766b428cb371abd406f6f0a0"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "10977cb7cbaa5ce86b78321f605b3d09"
server_key_iv: "43bbdc136f34bbb97c0ebc525b01d3d6"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.710947 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:40.722368 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:40.725117 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:40.725255 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:40.725713 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:40.726418 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:40.726461 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.726497 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:40.726513 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.735010 10694 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:45031
I20260926 12:28:40.735047 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:45031 every 8 connection(s)
I20260926 12:28:40.735843 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:40.740795 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10694
I20260926 12:28:40.740885 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:40.741178 7564 external_mini_cluster.cc:1428] Setting key 3abd569de18076c2415218354a711723
Sep 26 12:28:40 dist-test-slave-3jw2 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425720, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:40.749872 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:40.737485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49765 (local address 127.7.99.62:45923)
0926 12:28:40.737753 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:40.737758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:40.738402 (+ 644us) server_negotiation.cc:407] Connection header received
0926 12:28:40.739860 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:40.739863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:40.739910 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:40.739993 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:40.741231 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.741861 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.743047 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.743281 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.743387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:40.745848 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:40.745854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:40.745860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:40.745893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:40.747792 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:40.748312 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:40.748319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:40.748321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:40.748416 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:40.748733 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:40.748737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:40.748739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:40.748961 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:40.749085 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:40.749529 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 12:28:40.749660 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 12:28:40.750437 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:40.737850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45923 (local address 127.7.99.1:49765)
0926 12:28:40.738263 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:40.738301 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:40.739673 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:40.740140 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:40.740146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:40.740547 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:40.741089 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:40.741104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.741978 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.741981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:40.742923 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:40.742932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.743430 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.743434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:40.743601 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:40.744217 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:40.744241 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:40.745700 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:40.747905 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:40.747909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:40.747919 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:40.748210 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:40.748503 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:40.748505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:40.748507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:40.748628 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:40.749087 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:40.749091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:40.749292 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 12:28:40.749427 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 12:28:40.749653 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:28:40.751610 10823 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45923
I20260926 12:28:40.751865 10823 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:40.752372 10823 heartbeater.cc:511] Master 127.7.99.62:45923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:28:40.753854 10632 ts_manager.cc:194] Registered new tserver with Master: 2203bad9766b428cb371abd406f6f0a0 (127.7.99.1:45031)
I20260926 12:28:40.754918 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:49765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:40.768627 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:40.874926 10829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:40.875157 10829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:40.875197 10829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:40.879195 10829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:40.879244 10829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:40.879315 10829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:40.884322 10829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:40.885268 10829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:40.886231 10829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:40.895675 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:40.895670 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:40.895715 10829 server_base.cc:1034] running on GCE node
W20260926 12:28:40.895670 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:40.896306 10829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:40.896906 10829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:40.898080 10829 hybrid_clock.cc:648] HybridClock initialized: now 1790425720898056 us; error 47 us; skew 500 ppm
Sep 26 12:28:40 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425720, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:40.900733 10829 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:40.901501 10829 webserver.cc:460] Webserver started at http://127.7.99.2:36701/ using document root <none> and password file <none>
I20260926 12:28:40.902034 10829 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:40.902122 10829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:40.902336 10829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:40.904367 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "f3b3dbe733ba4d84b4e8b24d1828f891"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "f067fb137e205f2d6641b612327be8e6"
server_key_iv: "7fcbc3eddfb22c10b7940e50809488b0"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.904902 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "f3b3dbe733ba4d84b4e8b24d1828f891"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "f067fb137e205f2d6641b612327be8e6"
server_key_iv: "7fcbc3eddfb22c10b7940e50809488b0"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.908352 10829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:40.910564 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.911715 10829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:40.911846 10829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "f3b3dbe733ba4d84b4e8b24d1828f891"
format_stamp: "Formatted at 2026-09-26 12:28:40 on dist-test-slave-3jw2"
server_key: "f067fb137e205f2d6641b612327be8e6"
server_key_iv: "7fcbc3eddfb22c10b7940e50809488b0"
server_key_version: "encryptionkey@0"
I20260926 12:28:40.911945 10829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:40.923017 10829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:40.926087 10829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:40.926275 10829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:40.926811 10829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:40.927537 10829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:40.927613 10829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.927680 10829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:40.927719 10829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:40.936035 10829 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:34343
I20260926 12:28:40.936057 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:34343 every 8 connection(s)
I20260926 12:28:40.936911 10829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:40.944752 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10829
I20260926 12:28:40.944906 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:40.945194 7564 external_mini_cluster.cc:1428] Setting key da4dd139540a75074c6b9c381851c2cc
Sep 26 12:28:40 dist-test-slave-3jw2 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425720, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:40.950312 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:40.938590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:40173 (local address 127.7.99.62:45923)
0926 12:28:40.938742 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:40.938747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:40.939521 (+ 774us) server_negotiation.cc:407] Connection header received
0926 12:28:40.940958 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:40.940961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:40.941013 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:40.941087 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:40.942146 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.942649 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.943679 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.943885 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.943955 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:40.946605 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:40.946611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:40.946613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:40.946642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:40.948269 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:40.948723 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:40.948728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:40.948732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:40.948829 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:40.949130 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:40.949136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:40.949140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:40.949375 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:40.949475 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:40.949916 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 12:28:40.950068 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:28:40.950884 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:40.938968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45923 (local address 127.7.99.2:40173)
0926 12:28:40.939383 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:40.939423 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:40.940768 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:40.941239 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:40.941245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:40.941575 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:40.942018 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:40.942030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.942742 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.942744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:40.943586 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:40.943595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:40.943997 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:40.943999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:40.944139 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:40.944778 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:40.944793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:40.946459 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:40.948371 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:40.948376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:40.948386 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:40.948628 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:40.948920 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:40.948923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:40.948924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:40.949042 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:40.949467 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:40.949471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:40.949680 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 12:28:40.949822 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:28:40.950068 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260926 12:28:40.951970 10958 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45923
I20260926 12:28:40.952268 10958 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:40.952724 10958 heartbeater.cc:511] Master 127.7.99.62:45923 requested a full tablet report, sending...
I20260926 12:28:40.953768 10632 ts_manager.cc:194] Registered new tserver with Master: f3b3dbe733ba4d84b4e8b24d1828f891 (127.7.99.2:34343)
I20260926 12:28:40.954571 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:40173
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:40.972981 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:41.077472 10963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:41.077699 10963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:41.077739 10963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:41.081609 10963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:41.081650 10963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:41.081720 10963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:41.086617 10963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:41.087478 10963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:41.088377 10963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:41.097855 10969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:41.097893 10971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:41.097855 10968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:41.098150 10963 server_base.cc:1034] running on GCE node
I20260926 12:28:41.098804 10963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:41.099562 10963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:41.100731 10963 hybrid_clock.cc:648] HybridClock initialized: now 1790425721100705 us; error 49 us; skew 500 ppm
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425721, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:41.103328 10963 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:41.104228 10963 webserver.cc:460] Webserver started at http://127.7.99.3:46823/ using document root <none> and password file <none>
I20260926 12:28:41.104768 10963 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:41.104856 10963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:41.105089 10963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:41.107034 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "c1a586b268994b84ba3795ac46fb7338"
format_stamp: "Formatted at 2026-09-26 12:28:41 on dist-test-slave-3jw2"
server_key: "0b7476c56b50e76680d8f289f050a1f3"
server_key_iv: "afc6737ea36e0284a1b474fe482cb424"
server_key_version: "encryptionkey@0"
I20260926 12:28:41.107576 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "c1a586b268994b84ba3795ac46fb7338"
format_stamp: "Formatted at 2026-09-26 12:28:41 on dist-test-slave-3jw2"
server_key: "0b7476c56b50e76680d8f289f050a1f3"
server_key_iv: "afc6737ea36e0284a1b474fe482cb424"
server_key_version: "encryptionkey@0"
I20260926 12:28:41.111074 10963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:28:41.113309 10978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:41.114432 10963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:41.114596 10963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "c1a586b268994b84ba3795ac46fb7338"
format_stamp: "Formatted at 2026-09-26 12:28:41 on dist-test-slave-3jw2"
server_key: "0b7476c56b50e76680d8f289f050a1f3"
server_key_iv: "afc6737ea36e0284a1b474fe482cb424"
server_key_version: "encryptionkey@0"
I20260926 12:28:41.114698 10963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:41.129349 10963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:41.132603 10963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:41.132797 10963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:41.133358 10963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:41.134135 10963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:41.134202 10963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:41.134268 10963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:41.134316 10963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:41.142804 10963 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:37939
I20260926 12:28:41.142830 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:37939 every 8 connection(s)
I20260926 12:28:41.143637 10963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:41.148736 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 10963
I20260926 12:28:41.148859 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:41.149163 7564 external_mini_cluster.cc:1428] Setting key 215e5cef417acd4caaf2d8a3da7a8bd9
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425721, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:41.156747 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:41.145353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:51877 (local address 127.7.99.62:45923)
0926 12:28:41.145483 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:41.145486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:41.146456 (+ 970us) server_negotiation.cc:407] Connection header received
0926 12:28:41.147409 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:41.147412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:41.147481 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:41.147585 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:41.148877 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.149553 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.151042 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.151313 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.151392 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:41.153627 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:41.153630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:41.153632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:41.153653 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:41.155085 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:41.155498 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:41.155500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:41.155501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:41.155580 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:41.155845 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:41.155847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:41.155848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:41.155997 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:41.156065 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:41.156532 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 12:28:41.156631 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:28:41.157469 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:41.145702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45923 (local address 127.7.99.3:51877)
0926 12:28:41.146320 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:41.146357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:41.147209 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:41.147747 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:41.147754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:41.148103 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:41.148717 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:41.148738 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.149691 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.149695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:41.150895 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:41.150904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.151435 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.151438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:41.151596 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:41.152086 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:41.152100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:41.153493 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:41.155175 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:41.155178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:41.155189 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:41.155418 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:41.155655 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:41.155657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:41.155658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:41.155771 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:41.156077 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:41.156082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:41.156340 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 12:28:41.156486 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 12:28:41.156711 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":107,"threads_started":1}
I20260926 12:28:41.158433 11092 heartbeater.cc:347] Connected to a master server at 127.7.99.62:45923
I20260926 12:28:41.158628 11092 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:41.159034 11092 heartbeater.cc:511] Master 127.7.99.62:45923 requested a full tablet report, sending...
I20260926 12:28:41.159902 10632 ts_manager.cc:194] Registered new tserver with Master: c1a586b268994b84ba3795ac46fb7338 (127.7.99.3:37939)
I20260926 12:28:41.160496 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:51877
I20260926 12:28:41.162775 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:41.187528 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:41.179837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.7.99.62:45923)
0926 12:28:41.180019 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:41.180022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:41.180093 (+ 71us) server_negotiation.cc:407] Connection header received
0926 12:28:41.180264 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:41.180267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:41.180311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:41.180382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:41.181035 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.181539 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.182328 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:41.182524 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:41.182595 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:41.184909 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:41.184912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:41.184915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:41.184936 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:41.186139 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:41.186485 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:41.186488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:41.186489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:41.186555 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:41.186763 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:41.186765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:41.186766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:41.186957 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:41.187065 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:41.187231 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 12:28:41.187318 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 12:28:41.187917 10632 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57722
I20260926 12:28:41.188756 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10694
I20260926 12:28:41.194541 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10829
I20260926 12:28:41.199692 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10963
I20260926 12:28:41.205034 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 10600
2026-09-26T12:28:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2863 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[11104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[11104](info): set up 2 sockets
Sep 26 12:28:41 dist-test-slave-3jw2 krb5kdc[11104](info): commencing operation
krb5kdc: starting...
W20260926 12:28:42.236346 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:42.356626 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:37039
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:42.462545 11117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:42.462838 11117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:42.462917 11117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:42.466876 11117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:42.466964 11117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:42.467017 11117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:42.467056 11117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:42.467085 11117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:42.472083 11117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:42.473141 11117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:42.474318 11117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:42.482837 11125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:42.482847 11117 server_base.cc:1034] running on GCE node
W20260926 12:28:42.482885 11122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:42.482956 11123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:42.483574 11117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:42.484699 11117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:42.485857 11117 hybrid_clock.cc:648] HybridClock initialized: now 1790425722485835 us; error 32 us; skew 500 ppm
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:42.488476 11117 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:42.489341 11117 webserver.cc:460] Webserver started at http://127.7.99.62:36971/ using document root <none> and password file <none>
I20260926 12:28:42.489835 11117 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:42.489889 11117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:42.490059 11117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:42.491957 11117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "796e2690d162bfa942503d1cd25babf3"
server_key_iv: "5beefba6ab556e742ae49a51d6f43d78"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.492522 11117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "796e2690d162bfa942503d1cd25babf3"
server_key_iv: "5beefba6ab556e742ae49a51d6f43d78"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.495859 11117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:42.498159 11132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:42.499327 11117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:42.499449 11117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "796e2690d162bfa942503d1cd25babf3"
server_key_iv: "5beefba6ab556e742ae49a51d6f43d78"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.499549 11117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:42.538714 11117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:42.541622 11117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:42.541826 11117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:42.548455 11117 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:37039
I20260926 12:28:42.548497 11184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:37039 every 8 connection(s)
I20260926 12:28:42.549377 11117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:42.552134 11185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:42.553066 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11117
I20260926 12:28:42.553190 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:42.553486 7564 external_mini_cluster.cc:1428] Setting key 53440cbafb489583687a1736f87181d9
I20260926 12:28:42.558128 11185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3: Bootstrap starting.
I20260926 12:28:42.560276 11185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:42.560989 11185 log.cc:824] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:42.563055 11185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3: No bootstrap required, opened a new log
I20260926 12:28:42.566884 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:42.554897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.7.99.62:37039)
0926 12:28:42.555317 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:42.555331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:42.555364 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:28:42.555982 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:42.556000 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:42.556383 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:42.556712 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:42.557505 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.558501 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.559334 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.559793 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.559952 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:42.562495 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:42.562500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:42.562512 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:42.562534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:42.564252 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:42.564715 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:42.564719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:42.564723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:42.564821 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:42.565123 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:42.565125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:42.565126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:42.565458 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:42.565619 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:42.565899 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 12:28:42.566192 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20260926 12:28:42.567147 11185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } }
I20260926 12:28:42.567356 11185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:42.567425 11185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e313a88bbde455cb4ceb1f7cd0ed6a3, State: Initialized, Role: FOLLOWER
I20260926 12:28:42.567840 11185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } }
I20260926 12:28:42.568004 11185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:42.568094 11185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:42.568223 11185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:42.569025 11185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } }
I20260926 12:28:42.569350 11185 leader_election.cc:302] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e313a88bbde455cb4ceb1f7cd0ed6a3; no voters:
I20260926 12:28:42.569639 11185 leader_election.cc:288] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:42.569743 11190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:42.569976 11190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 8e313a88bbde455cb4ceb1f7cd0ed6a3, State: Running, Role: LEADER
I20260926 12:28:42.570247 11190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } }
I20260926 12:28:42.570788 11185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:42.571856 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } } }
I20260926 12:28:42.571995 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:42.571874 11192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e313a88bbde455cb4ceb1f7cd0ed6a3. Latest consensus state: current_term: 1 leader_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e313a88bbde455cb4ceb1f7cd0ed6a3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37039 } } }
I20260926 12:28:42.572105 11192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:42.572480 11199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:42.575474 11199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:42.580778 11199 catalog_manager.cc:1347] Generated new cluster ID: 95ce3b8f931547f3b516d748d59bfd84
I20260926 12:28:42.580840 11199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:42.592576 11199 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:42.593459 11199 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:42.602495 11199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e313a88bbde455cb4ceb1f7cd0ed6a3: Generated new TSK 0
I20260926 12:28:42.603189 11199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:42.634609 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:42.751801 11211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:42.752072 11211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:42.752116 11211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:42.756130 11211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:42.756196 11211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:42.756287 11211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:42.761360 11211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:42.762377 11211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:42.763370 11211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:42.772811 11219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:42.772924 11211 server_base.cc:1034] running on GCE node
W20260926 12:28:42.772841 11217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:42.772842 11216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:42.773377 11211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:42.773979 11211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:42.775156 11211 hybrid_clock.cc:648] HybridClock initialized: now 1790425722775142 us; error 25 us; skew 500 ppm
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:42.777761 11211 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:42.778549 11211 webserver.cc:460] Webserver started at http://127.7.99.1:40813/ using document root <none> and password file <none>
I20260926 12:28:42.779080 11211 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:42.779165 11211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:42.779376 11211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:42.781337 11211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "811ffddca0d34dfd949185428579c700"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "17b55bf3d907f0159df04dbba84ac442"
server_key_iv: "636fe4f5de105b250ceaab562dd722ed"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.781893 11211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "811ffddca0d34dfd949185428579c700"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "17b55bf3d907f0159df04dbba84ac442"
server_key_iv: "636fe4f5de105b250ceaab562dd722ed"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.785296 11211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:42.787509 11226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:42.788688 11211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:42.788806 11211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "811ffddca0d34dfd949185428579c700"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "17b55bf3d907f0159df04dbba84ac442"
server_key_iv: "636fe4f5de105b250ceaab562dd722ed"
server_key_version: "encryptionkey@0"
I20260926 12:28:42.788908 11211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:42.808641 11211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:42.811314 11211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:42.811511 11211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:42.812085 11211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:42.812912 11211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:42.812980 11211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:42.813071 11211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:42.813117 11211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:42.821628 11211 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:39575
I20260926 12:28:42.821657 11339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:39575 every 8 connection(s)
I20260926 12:28:42.822491 11211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:42.831175 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11211
I20260926 12:28:42.831312 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:42.831630 7564 external_mini_cluster.cc:1428] Setting key 3d9f71d9f32dda3fb7da67918260ee68
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:42.836966 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:42.824250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:33519 (local address 127.7.99.62:37039)
0926 12:28:42.824384 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:42.824388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:42.825521 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 12:28:42.827057 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:42.827060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:42.827108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:42.827186 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:42.828430 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.828964 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.830140 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.830356 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.830437 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:42.833283 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:42.833287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:42.833290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:42.833315 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:42.834780 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:42.835445 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:42.835449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:42.835451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:42.835531 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:42.835876 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:42.835879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:42.835881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:42.836065 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:42.836172 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:42.836674 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 12:28:42.836796 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:28:42.837759 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:42.824800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37039 (local address 127.7.99.1:33519)
0926 12:28:42.825314 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:42.825367 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:42.826849 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:42.827336 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:42.827343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:42.827732 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:42.828275 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:42.828291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.829098 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.829102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:42.830035 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:42.830045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:42.830479 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:42.830482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:42.830623 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:42.831179 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:42.831204 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:42.833112 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:42.835028 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:42.835034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:42.835050 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:42.835335 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:42.835630 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:42.835633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:42.835635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:42.835776 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:42.836184 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:42.836191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:42.836433 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 12:28:42.836619 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 12:28:42.836946 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260926 12:28:42.838940 11340 heartbeater.cc:347] Connected to a master server at 127.7.99.62:37039
I20260926 12:28:42.839192 11340 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:42.839654 11340 heartbeater.cc:511] Master 127.7.99.62:37039 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:28:42.841343 11149 ts_manager.cc:194] Registered new tserver with Master: 811ffddca0d34dfd949185428579c700 (127.7.99.1:39575)
I20260926 12:28:42.842546 11149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:33519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:42.859119 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:42.972301 11345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:42.972602 11345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:42.972679 11345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:42.976714 11345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:42.976801 11345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:42.976910 11345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:42.982069 11345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:42.983090 11345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:42.984126 11345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:42.993466 11351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:42.993481 11350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:42.993578 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:42.993789 11345 server_base.cc:1034] running on GCE node
I20260926 12:28:42.994240 11345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:42.994834 11345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:42.996008 11345 hybrid_clock.cc:648] HybridClock initialized: now 1790425722995996 us; error 23 us; skew 500 ppm
Sep 26 12:28:42 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:42.998751 11345 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:42.999495 11345 webserver.cc:460] Webserver started at http://127.7.99.2:39533/ using document root <none> and password file <none>
I20260926 12:28:43.000027 11345 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:43.000114 11345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:43.000376 11345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:43.002271 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "6d74a56e0650466793bdc6ebc4afe9e7"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "22c94cee653e50db8d82b5fe6ec4773f"
server_key_iv: "9e95800bb86cff9f9fd6ec71b7229352"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.002928 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "6d74a56e0650466793bdc6ebc4afe9e7"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "22c94cee653e50db8d82b5fe6ec4773f"
server_key_iv: "9e95800bb86cff9f9fd6ec71b7229352"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.006629 11345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 12:28:43.009016 11360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.010170 11345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:28:43.010294 11345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "6d74a56e0650466793bdc6ebc4afe9e7"
format_stamp: "Formatted at 2026-09-26 12:28:42 on dist-test-slave-3jw2"
server_key: "22c94cee653e50db8d82b5fe6ec4773f"
server_key_iv: "9e95800bb86cff9f9fd6ec71b7229352"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.010396 11345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:43.036561 11345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:43.039201 11345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:43.039387 11345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:43.039973 11345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:43.040771 11345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:43.040839 11345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.040906 11345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:43.040956 11345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.049458 11345 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:45759
I20260926 12:28:43.049491 11473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:45759 every 8 connection(s)
I20260926 12:28:43.050297 11345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:43.055599 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11345
I20260926 12:28:43.055734 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:43.055997 7564 external_mini_cluster.cc:1428] Setting key 08e366c44f147af1a7a89fd444ee5d15
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425722, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:43.064237 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.052038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:36905 (local address 127.7.99.62:37039)
0926 12:28:43.052212 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:43.052217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:43.052978 (+ 761us) server_negotiation.cc:407] Connection header received
0926 12:28:43.054231 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:43.054234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:43.054277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:43.054347 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:43.055507 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.056198 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.057530 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.057762 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.057868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.060603 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:43.060606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:43.060608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:43.060642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:43.062148 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.062743 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.062745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.062746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.062812 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.063171 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.063173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.063174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.063322 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:43.063391 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:43.063958 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 12:28:43.064074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:43.065328 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.052437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37039 (local address 127.7.99.2:36905)
0926 12:28:43.052838 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:43.052877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:43.054056 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:43.054502 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:43.054508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:43.054864 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:43.055336 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:43.055348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.056330 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.056333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:43.057410 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:43.057419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.057898 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.057902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:43.058068 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.058646 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:43.058660 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:43.060427 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:43.062260 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:43.062267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:43.062283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:43.062650 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:43.062899 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:43.062904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:43.062907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:43.063075 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:43.063420 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:43.063426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:43.063718 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 12:28:43.063919 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 12:28:43.064294 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:28:43.066560 11474 heartbeater.cc:347] Connected to a master server at 127.7.99.62:37039
I20260926 12:28:43.066789 11474 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:43.067188 11474 heartbeater.cc:511] Master 127.7.99.62:37039 requested a full tablet report, sending...
I20260926 12:28:43.068233 11149 ts_manager.cc:194] Registered new tserver with Master: 6d74a56e0650466793bdc6ebc4afe9e7 (127.7.99.2:45759)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:28:43.068864 11149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:36905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:43.083144 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:43.191841 11479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:43.192123 11479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:43.192236 11479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:43.196244 11479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:43.196288 11479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:43.196388 11479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:43.201373 11479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:43.202353 11479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:43.203435 11479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:43.212489 11487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:43.212685 11479 server_base.cc:1034] running on GCE node
W20260926 12:28:43.212481 11485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:43.212584 11484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:43.213169 11479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:43.213790 11479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:43.214956 11479 hybrid_clock.cc:648] HybridClock initialized: now 1790425723214918 us; error 55 us; skew 500 ppm
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:43.217660 11479 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:43.218452 11479 webserver.cc:460] Webserver started at http://127.7.99.3:34861/ using document root <none> and password file <none>
I20260926 12:28:43.219014 11479 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:43.219100 11479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:43.219328 11479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:43.221338 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "b2d4a9df1cc642188e8ad541e1188fff"
format_stamp: "Formatted at 2026-09-26 12:28:43 on dist-test-slave-3jw2"
server_key: "ae7ca35339f5e63e99af3e1bea023b6b"
server_key_iv: "e706789b9a21fd8965927637386c2129"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.221916 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d4a9df1cc642188e8ad541e1188fff"
format_stamp: "Formatted at 2026-09-26 12:28:43 on dist-test-slave-3jw2"
server_key: "ae7ca35339f5e63e99af3e1bea023b6b"
server_key_iv: "e706789b9a21fd8965927637386c2129"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.225476 11479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:43.227738 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.228981 11479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:43.229106 11479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "b2d4a9df1cc642188e8ad541e1188fff"
format_stamp: "Formatted at 2026-09-26 12:28:43 on dist-test-slave-3jw2"
server_key: "ae7ca35339f5e63e99af3e1bea023b6b"
server_key_iv: "e706789b9a21fd8965927637386c2129"
server_key_version: "encryptionkey@0"
I20260926 12:28:43.229208 11479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:43.260016 11479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:43.262722 11479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:43.262910 11479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:43.263458 11479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:43.264308 11479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:43.264374 11479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.264437 11479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:43.264494 11479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:43.273712 11479 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:46011
I20260926 12:28:43.273784 11607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:46011 every 8 connection(s)
I20260926 12:28:43.274660 11479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:43.279048 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11479
I20260926 12:28:43.279183 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:43.279462 7564 external_mini_cluster.cc:1428] Setting key 8456897913dfcc14b3851431c0281141
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:43.287685 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.276431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:46775 (local address 127.7.99.62:37039)
0926 12:28:43.276561 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:43.276565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:43.277366 (+ 801us) server_negotiation.cc:407] Connection header received
0926 12:28:43.278764 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:43.278767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:43.278822 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:43.278911 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:43.280251 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.280812 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.281936 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.282160 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.282272 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.284562 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:43.284570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:43.284572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:43.284593 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:43.285732 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.286218 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.286223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.286225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.286303 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.286636 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.286641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.286643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.286816 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:43.286924 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:43.287394 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 12:28:43.287519 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:28:43.288486 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.276829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37039 (local address 127.7.99.3:46775)
0926 12:28:43.277229 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:43.277268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:43.278566 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:43.279104 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:43.279110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:43.279506 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:43.280091 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:43.280107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.280931 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.280934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:43.281814 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:43.281822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.282300 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.282304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:43.282468 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.282956 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:43.282971 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:43.284410 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:43.285845 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:43.285851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:43.285864 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:43.286116 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:43.286403 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:43.286405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:43.286407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:43.286542 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:43.286919 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:43.286926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:43.287162 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 12:28:43.287340 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 12:28:43.287639 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260926 12:28:43.289471 11608 heartbeater.cc:347] Connected to a master server at 127.7.99.62:37039
I20260926 12:28:43.289702 11608 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:43.290146 11608 heartbeater.cc:511] Master 127.7.99.62:37039 requested a full tablet report, sending...
I20260926 12:28:43.291076 11149 ts_manager.cc:194] Registered new tserver with Master: b2d4a9df1cc642188e8ad541e1188fff (127.7.99.3:46011)
I20260926 12:28:43.291679 11149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:46775
I20260926 12:28:43.293125 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:43.318503 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.308980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.7.99.1:39575)
0926 12:28:43.309268 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:43.309281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:43.309319 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:28:43.309366 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:43.309370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:43.309546 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:43.309653 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:43.310259 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.311021 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.311789 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.312130 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.312277 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.314627 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:43.314636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:43.314648 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:43.314670 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:43.316700 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.317132 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.317138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.317140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.317227 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.317507 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.317511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.317513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.317746 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:43.317955 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:43.318176 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 12:28:43.318325 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:43.328763 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:43.321232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.7.99.62:37039)
0926 12:28:43.321347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:43.321358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:43.321371 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:28:43.321451 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:43.321453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:43.321496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:43.321560 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:43.322080 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.322580 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.323285 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:43.323568 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:43.323644 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:43.325765 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:43.325768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:43.325770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:43.325792 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:43.327124 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.327529 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.327533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.327542 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.327619 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:43.327877 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:43.327881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:43.327883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:43.328095 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:43.328280 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:43.328484 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 12:28:43.328604 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 12:28:43.329129 11149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56380
I20260926 12:28:43.329771 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11211
I20260926 12:28:43.336189 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11345
I20260926 12:28:43.342691 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11479
I20260926 12:28:43.347630 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11117
2026-09-26T12:28:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2142 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11620](info): set up 2 sockets
Sep 26 12:28:43 dist-test-slave-3jw2 krb5kdc[11620](info): commencing operation
krb5kdc: starting...
W20260926 12:28:44.384541 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:44 dist-test-slave-3jw2 krb5kdc[11620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:44.503225 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46027
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:44.615094 11633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:44.615414 11633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:44.615641 11633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:44.619968 11633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:44.620065 11633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:44.620108 11633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:44.620168 11633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:44.620224 11633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:44.625445 11633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:46027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:44.626533 11633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:44.627799 11633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:44.635880 11638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:44.636051 11633 server_base.cc:1034] running on GCE node
W20260926 12:28:44.635890 11641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:44.635919 11639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:44.636775 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:44.637768 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:44.638944 11633 hybrid_clock.cc:648] HybridClock initialized: now 1790425724638926 us; error 30 us; skew 500 ppm
Sep 26 12:28:44 dist-test-slave-3jw2 krb5kdc[11620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425724, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:44.641727 11633 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:44.642439 11633 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:44.642529 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:44.642748 11633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:44.644765 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "251cd6fa8a07488186e1ece74a2d9de9"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "a97f9bf770ff5c14b4ea26a06b0b78ed"
server_key_iv: "b06f980ab242e00b59debd174f8fcb3d"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.645318 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "251cd6fa8a07488186e1ece74a2d9de9"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "a97f9bf770ff5c14b4ea26a06b0b78ed"
server_key_iv: "b06f980ab242e00b59debd174f8fcb3d"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.648917 11633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:44.651230 11647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:44.652432 11633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:44.652573 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "251cd6fa8a07488186e1ece74a2d9de9"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "a97f9bf770ff5c14b4ea26a06b0b78ed"
server_key_iv: "b06f980ab242e00b59debd174f8fcb3d"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.652696 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:44.663851 11633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:44.666725 11633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:44.666922 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:44.673763 11633 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:46027
I20260926 12:28:44.674026 11699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:46027 every 8 connection(s)
I20260926 12:28:44.674417 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:44.677119 11700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:44.679219 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11633
I20260926 12:28:44.679355 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:44.679623 7564 external_mini_cluster.cc:1428] Setting key 8355b1dd5ad5763e9ec00c8a412152c7
I20260926 12:28:44.682967 11700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: Bootstrap starting.
I20260926 12:28:44.685499 11700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:44.686264 11700 log.cc:824] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:28:44 dist-test-slave-3jw2 krb5kdc[11620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:44.688367 11700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: No bootstrap required, opened a new log
I20260926 12:28:44.692559 11700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } }
I20260926 12:28:44.692759 11700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:44.692816 11700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251cd6fa8a07488186e1ece74a2d9de9, State: Initialized, Role: FOLLOWER
I20260926 12:28:44.693228 11700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } }
I20260926 12:28:44.693341 11700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:44.693425 11700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:44.693511 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:44.680888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.7.99.62:46027)
0926 12:28:44.681240 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:44.681261 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:44.681305 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:28:44.681937 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:44.681952 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:44.682272 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:44.682560 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:44.683438 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.684464 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.685634 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.686106 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.686222 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:44.688898 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:44.688903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:44.688915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:44.688939 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:44.690464 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:44.690965 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:44.690969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:44.690974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:44.691086 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:44.691354 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:44.691356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:44.691357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:44.691710 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:44.691867 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:44.692232 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 12:28:44.692552 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
I20260926 12:28:44.693552 11700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:44.694392 11700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } }
I20260926 12:28:44.694720 11700 leader_election.cc:302] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 251cd6fa8a07488186e1ece74a2d9de9; no voters:
I20260926 12:28:44.695003 11700 leader_election.cc:288] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:44.695102 11705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:44.695576 11705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [term 1 LEADER]: Becoming Leader. State: Replica: 251cd6fa8a07488186e1ece74a2d9de9, State: Running, Role: LEADER
I20260926 12:28:44.695976 11705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } }
I20260926 12:28:44.696364 11700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:44.697182 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251cd6fa8a07488186e1ece74a2d9de9. Latest consensus state: current_term: 1 leader_uuid: "251cd6fa8a07488186e1ece74a2d9de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } } }
I20260926 12:28:44.697297 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:44.697896 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "251cd6fa8a07488186e1ece74a2d9de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251cd6fa8a07488186e1ece74a2d9de9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46027 } } }
I20260926 12:28:44.697980 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:44.698210 11714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:44.701282 11714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:44.706625 11714 catalog_manager.cc:1347] Generated new cluster ID: 2ec30e5c3d8c411ab53940bf6b2b15b5
I20260926 12:28:44.706694 11714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:44.714447 11714 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:44.715337 11714 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:44.721688 11714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: Generated new TSK 0
I20260926 12:28:44.722493 11714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:44.750584 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:44.859603 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:44.859848 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:44.859892 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:44.864020 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:44.864091 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:44.864167 11726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:44.869457 11726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:44.870435 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:44.871443 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:44.880967 11731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:44.880999 11734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:44.881019 11732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:44.881345 11726 server_base.cc:1034] running on GCE node
I20260926 12:28:44.881788 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:44.882444 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:44.883603 11726 hybrid_clock.cc:648] HybridClock initialized: now 1790425724883575 us; error 38 us; skew 500 ppm
Sep 26 12:28:44 dist-test-slave-3jw2 krb5kdc[11620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425724, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:44.886405 11726 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:44.887147 11726 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:44.887240 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:44.887473 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:44.889569 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "31d1e74b98154075961687c0206d528b"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "ab84601e21fad7752dd6e594043d43f7"
server_key_iv: "e9342dbe31d097117dc6445679a364cf"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.890125 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "31d1e74b98154075961687c0206d528b"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "ab84601e21fad7752dd6e594043d43f7"
server_key_iv: "e9342dbe31d097117dc6445679a364cf"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.893769 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:28:44.896091 11740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:44.897431 11726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:44.897563 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "31d1e74b98154075961687c0206d528b"
format_stamp: "Formatted at 2026-09-26 12:28:44 on dist-test-slave-3jw2"
server_key: "ab84601e21fad7752dd6e594043d43f7"
server_key_iv: "e9342dbe31d097117dc6445679a364cf"
server_key_version: "encryptionkey@0"
I20260926 12:28:44.897688 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:44.940624 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:44.943516 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:44.943732 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:44.944350 11726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:44.945489 11726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:44.945550 11726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:44.945631 11726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:44.945663 11726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:44.954272 11726 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:41059
I20260926 12:28:44.954285 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:41059 every 8 connection(s)
I20260926 12:28:44.954861 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:44.956838 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11726
I20260926 12:28:44.956957 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:44.957262 7564 external_mini_cluster.cc:1428] Setting key 81ae4a340bd0fd5f07fccfbe2e1769dd
Sep 26 12:28:44 dist-test-slave-3jw2 krb5kdc[11620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425724, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:28:44.969471 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:44.956727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42963 (local address 127.7.99.62:46027)
0926 12:28:44.956887 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:44.956891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:44.957998 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 12:28:44.959622 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:44.959625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:44.959675 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:44.959754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:44.961425 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.961937 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.963157 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.963389 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.963492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:44.966093 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:44.966096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:44.966098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:44.966120 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:44.967408 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:44.967918 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:44.967921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:44.967922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:44.967990 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:44.968418 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:44.968422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:44.968424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:44.968598 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:44.968705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:44.969256 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 12:28:44.969355 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:28:44.970620 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:44.957246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46027 (local address 127.7.99.1:42963)
0926 12:28:44.957824 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:44.957881 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:44.959408 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:44.959928 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:44.959957 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:44.960559 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:44.961294 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:44.961317 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.962066 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.962070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:44.963033 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:44.963043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:44.963536 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:44.963549 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:44.963718 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:44.964327 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:44.964343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:44.965925 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:44.967523 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:44.967528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:44.967551 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:44.967787 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:44.968136 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:44.968138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:44.968140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:44.968320 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:44.968704 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:44.968711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:44.968981 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 12:28:44.969188 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 12:28:44.969559 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260926 12:28:44.971880 11854 heartbeater.cc:347] Connected to a master server at 127.7.99.62:46027
I20260926 12:28:44.972141 11854 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:44.972597 11854 heartbeater.cc:511] Master 127.7.99.62:46027 requested a full tablet report, sending...
I20260926 12:28:44.974215 11664 ts_manager.cc:194] Registered new tserver with Master: 31d1e74b98154075961687c0206d528b (127.7.99.1:41059)
I20260926 12:28:44.975889 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:42963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:44.985016 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:45.091791 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:45.092058 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:45.092128 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:45.095952 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:45.096048 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:45.096177 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:45.101392 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:45.102451 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:45.103436 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:45.112512 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.112550 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.112874 11859 server_base.cc:1034] running on GCE node
W20260926 12:28:45.113049 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.113548 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:45.114306 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:45.115490 11859 hybrid_clock.cc:648] HybridClock initialized: now 1790425725115453 us; error 29 us; skew 500 ppm
Sep 26 12:28:45 dist-test-slave-3jw2 krb5kdc[11620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425725, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:45.118148 11859 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:45.118824 11859 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:45.118912 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:45.119139 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:45.121182 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "baeb86753662469299f9c94f0f4b3865"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "d93c3a6db61599e5d5681aeae15c8c89"
server_key_iv: "0830f06385bee522365da9608476f1d1"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.121731 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "baeb86753662469299f9c94f0f4b3865"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "d93c3a6db61599e5d5681aeae15c8c89"
server_key_iv: "0830f06385bee522365da9608476f1d1"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.125295 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:45.127548 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.128727 11859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:45.128858 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "baeb86753662469299f9c94f0f4b3865"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "d93c3a6db61599e5d5681aeae15c8c89"
server_key_iv: "0830f06385bee522365da9608476f1d1"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.128962 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:45.140858 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:45.143501 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:45.143695 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:45.144263 11859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:45.145044 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:45.145113 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.145182 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:45.145223 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.153769 11859 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:32865
I20260926 12:28:45.153841 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:32865 every 8 connection(s)
I20260926 12:28:45.154364 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:45.160696 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11859
I20260926 12:28:45.160828 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:45.161108 7564 external_mini_cluster.cc:1428] Setting key f31610479c3fb3cfff4230c0cb76a6a3
Sep 26 12:28:45 dist-test-slave-3jw2 krb5kdc[11620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425725, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:45.168576 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.156077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:37791 (local address 127.7.99.62:46027)
0926 12:28:45.156246 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.156251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.157067 (+ 816us) server_negotiation.cc:407] Connection header received
0926 12:28:45.158488 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.158491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.158557 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.158634 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:45.159813 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.160429 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.161723 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.162014 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.162119 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.164684 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:45.164690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:45.164696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:45.164729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:45.166268 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.166921 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.166923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.166924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.167002 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.167404 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.167406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.167407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.167575 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:45.167645 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.168349 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.168452 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:45.169865 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.156486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46027 (local address 127.7.99.2:37791)
0926 12:28:45.156916 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.156958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.158270 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.158802 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.158809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.159184 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:45.159655 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.159669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.160552 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.160558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.161569 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.161579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.162153 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.162156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.162327 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.163001 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:45.163017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:45.164506 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:45.166385 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:45.166394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:45.166414 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:45.166824 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:45.167091 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:45.167094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:45.167098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:45.167299 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:45.167679 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:45.167688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:45.168045 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.168323 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.168720 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:28:45.171120 11987 heartbeater.cc:347] Connected to a master server at 127.7.99.62:46027
I20260926 12:28:45.171380 11987 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:45.171809 11987 heartbeater.cc:511] Master 127.7.99.62:46027 requested a full tablet report, sending...
I20260926 12:28:45.172943 11664 ts_manager.cc:194] Registered new tserver with Master: baeb86753662469299f9c94f0f4b3865 (127.7.99.2:32865)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:28:45.173738 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:37791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:45.189249 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:45.299110 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:45.299458 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:45.299553 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:45.303705 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:45.303790 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:45.303879 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:45.309245 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:45.310257 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:45.311278 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:45.320523 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.320533 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.320549 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.320809 11992 server_base.cc:1034] running on GCE node
I20260926 12:28:45.321241 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:45.321892 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:45.323083 11992 hybrid_clock.cc:648] HybridClock initialized: now 1790425725323068 us; error 27 us; skew 500 ppm
Sep 26 12:28:45 dist-test-slave-3jw2 krb5kdc[11620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425725, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:45.325764 11992 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:45.326468 11992 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:45.326560 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:45.326788 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:45.328892 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "668dc2bd05a54150a0aa75964475ca96"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "f83d3db14764c195a76af77acb76d929"
server_key_iv: "076bad9f1720db6164c71754a8272639"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.329454 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "668dc2bd05a54150a0aa75964475ca96"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "f83d3db14764c195a76af77acb76d929"
server_key_iv: "076bad9f1720db6164c71754a8272639"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.333146 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:45.335412 12006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.336705 11992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:45.336853 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "668dc2bd05a54150a0aa75964475ca96"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "f83d3db14764c195a76af77acb76d929"
server_key_iv: "076bad9f1720db6164c71754a8272639"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.336962 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:45.352669 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:45.355450 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:45.355671 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:45.356329 11992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:45.357204 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:45.357271 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.357342 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:45.357383 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.365962 11992 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33393
I20260926 12:28:45.365981 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33393 every 8 connection(s)
I20260926 12:28:45.366694 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:45.375613 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 11992
I20260926 12:28:45.375737 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:45.376035 7564 external_mini_cluster.cc:1428] Setting key d217179b6d4eebbf8d40dd50e15cf303
Sep 26 12:28:45 dist-test-slave-3jw2 krb5kdc[11620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425725, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:45.381193 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.368862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:43645 (local address 127.7.99.62:46027)
0926 12:28:45.369014 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.369017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.369823 (+ 806us) server_negotiation.cc:407] Connection header received
0926 12:28:45.371216 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.371219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.371280 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.371359 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:45.372555 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.373081 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.374251 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.374479 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.374600 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.377543 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:45.377547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:45.377549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:45.377573 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:45.379109 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.379638 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.379641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.379643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.379725 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.380084 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.380088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.380089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.380327 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:45.380427 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.380907 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.381033 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:28:45.381949 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.369234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46027 (local address 127.7.99.3:43645)
0926 12:28:45.369663 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.369704 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.370988 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.371535 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.371542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.371891 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:45.372400 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.372415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.373207 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.373211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.374106 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.374115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.374637 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.374641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.374819 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.375527 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:45.375547 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:45.377350 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:45.379248 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:45.379253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:45.379267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:45.379525 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:45.379825 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:45.379828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:45.379830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:45.379981 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:45.380436 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:45.380445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:45.380684 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.380869 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.381176 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260926 12:28:45.382965 12120 heartbeater.cc:347] Connected to a master server at 127.7.99.62:46027
I20260926 12:28:45.383220 12120 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:45.383646 12120 heartbeater.cc:511] Master 127.7.99.62:46027 requested a full tablet report, sending...
I20260926 12:28:45.384703 11664 ts_manager.cc:194] Registered new tserver with Master: 668dc2bd05a54150a0aa75964475ca96 (127.7.99.3:33393)
I20260926 12:28:45.385219 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:43645
I20260926 12:28:45.389743 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:28:45.398674 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.391506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.7.99.62:46027)
0926 12:28:45.391629 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.391632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.391818 (+ 186us) server_negotiation.cc:407] Connection header received
0926 12:28:45.391906 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.391908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.391941 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.392008 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:45.392695 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.393195 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.393976 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.394187 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.394263 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.395446 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:45.395449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:45.395451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:45.395471 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:45.397102 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.397516 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.397519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.397520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.397590 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:45.397842 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:45.397844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:45.397845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:45.398085 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:45.398184 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.398401 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.398488 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:28:45.401230 11664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:45.404381 11664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:45.417930 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.412990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.7.99.3:33393)
0926 12:28:45.413368 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.413379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.413400 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:28:45.413477 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.413482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.413677 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.413831 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:45.414136 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.414918 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.416485 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.417245 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417365 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.417528 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.417689 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.417779 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":101,"threads_started":1}
I20260926 12:28:45.417923 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.412251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33393 (local address 127.0.0.1:49902)
0926 12:28:45.412633 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.412681 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.412806 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.413840 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.413849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.413870 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:45.414019 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.414028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.415086 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.415105 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.416360 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.416370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.417363 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.417504 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.417545 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.417655 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.417776 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":69,"threads_started":1}
I20260926 12:28:45.418291 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.412526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:41059 (local address 127.0.0.1:37464)
0926 12:28:45.413025 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.413036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.413114 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.414002 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.414005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.414020 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:45.414128 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.414133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.415599 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.415603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.416726 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.416734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.417903 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.417988 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.418008 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.418077 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.418145 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":75,"thread_start_us":85,"threads_started":1}
I20260926 12:28:45.418710 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.413015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.7.99.1:41059)
0926 12:28:45.413431 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.413437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.413452 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:28:45.413510 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.413514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.413677 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.413831 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:45.414258 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.415457 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.416836 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.417604 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417710 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.417839 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.418455 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.418568 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":85,"threads_started":1}
I20260926 12:28:45.418808 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.412295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.7.99.2:32865)
0926 12:28:45.413259 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.413271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.413439 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:28:45.413534 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.413539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.413738 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.413893 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:45.414301 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.415457 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417051 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.417824 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.417921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.418061 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.418565 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.418661 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":112,"threads_started":1}
I20260926 12:28:45.418478 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.412171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32865 (local address 127.0.0.1:58742)
0926 12:28:45.413267 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.413279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.413352 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.414045 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.414050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.414070 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:45.414162 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.414167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.415628 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.415632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.416924 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.416934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.418118 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.418121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.418194 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.418213 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.418276 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.418346 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":59,"thread_start_us":114,"threads_started":1}
I20260926 12:28:45.421033 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda (DEFAULT_TABLE table=test-table [id=140e939e69db4b03a3b344fcc9a84d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:45.421397 11788 tablet_service.cc:1467] Processing CreateTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda (DEFAULT_TABLE table=test-table [id=140e939e69db4b03a3b344fcc9a84d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:45.421605 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda (DEFAULT_TABLE table=test-table [id=140e939e69db4b03a3b344fcc9a84d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:45.422149 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e2c016ff1e49d1b9de6fb1b105cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:45.422359 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e2c016ff1e49d1b9de6fb1b105cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:45.422484 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e2c016ff1e49d1b9de6fb1b105cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:45.428139 12137 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Bootstrap starting.
I20260926 12:28:45.429663 12139 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Bootstrap starting.
I20260926 12:28:45.430881 12137 tablet_bootstrap.cc:654] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:45.431324 12139 tablet_bootstrap.cc:654] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:45.432010 12137 log.cc:824] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:45.432487 12138 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: Bootstrap starting.
I20260926 12:28:45.432467 12139 log.cc:824] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:45.434129 12137 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: No bootstrap required, opened a new log
I20260926 12:28:45.434351 12139 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: No bootstrap required, opened a new log
I20260926 12:28:45.434540 12138 tablet_bootstrap.cc:654] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:45.434589 12137 ts_tablet_manager.cc:1397] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260926 12:28:45.434628 12139 ts_tablet_manager.cc:1397] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:45.435700 12138 log.cc:824] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:45.437553 12138 tablet_bootstrap.cc:492] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: No bootstrap required, opened a new log
I20260926 12:28:45.437722 12138 ts_tablet_manager.cc:1397] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:45.437736 12137 raft_consensus.cc:348] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.438022 12137 raft_consensus.cc:374] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:45.438095 12137 raft_consensus.cc:729] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d1e74b98154075961687c0206d528b, State: Initialized, Role: FOLLOWER
I20260926 12:28:45.438390 12139 raft_consensus.cc:348] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.438568 12137 consensus_queue.cc:260] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.438673 12139 raft_consensus.cc:374] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:45.438753 12139 raft_consensus.cc:729] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668dc2bd05a54150a0aa75964475ca96, State: Initialized, Role: FOLLOWER
I20260926 12:28:45.439172 12139 consensus_queue.cc:260] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.439570 12137 ts_tablet_manager.cc:1428] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:45.439813 12120 heartbeater.cc:503] Master 127.7.99.62:46027 was elected leader, sending a full tablet report...
I20260926 12:28:45.439874 11854 heartbeater.cc:503] Master 127.7.99.62:46027 was elected leader, sending a full tablet report...
I20260926 12:28:45.440068 12139 ts_tablet_manager.cc:1428] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260926 12:28:45.441350 12138 raft_consensus.cc:348] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.441618 12138 raft_consensus.cc:374] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:45.441697 12138 raft_consensus.cc:729] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeb86753662469299f9c94f0f4b3865, State: Initialized, Role: FOLLOWER
I20260926 12:28:45.442130 12138 consensus_queue.cc:260] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.443058 12138 ts_tablet_manager.cc:1428] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:28:45.443310 11987 heartbeater.cc:503] Master 127.7.99.62:46027 was elected leader, sending a full tablet report...
I20260926 12:28:45.448045 12144 raft_consensus.cc:482] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:45.448228 12144 raft_consensus.cc:504] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.449152 12144 leader_election.cc:288] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baeb86753662469299f9c94f0f4b3865 (127.7.99.2:32865), 31d1e74b98154075961687c0206d528b (127.7.99.1:41059)
I20260926 12:28:45.453128 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.449320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32865 (local address 127.7.99.3:47535)
0926 12:28:45.449533 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.449549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.449633 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.449922 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.449924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.449937 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:45.450077 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.450082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.451090 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.451093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.451909 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.451918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.452807 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.452888 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.452908 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.452967 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.453019 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 12:28:45.453217 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.449518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:47535 (local address 127.7.99.2:32865)
0926 12:28:45.449633 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.449639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.449653 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:28:45.449778 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.449781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.449829 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.449912 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:45.450203 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.450997 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452037 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.452684 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.452852 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.453007 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.453081 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:28:45.453372 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.449372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:41059 (local address 127.7.99.3:58841)
0926 12:28:45.449814 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.449825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.449931 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.450142 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.450146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.450167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:45.450269 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.450273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.451188 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.451193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.451992 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:45.452004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.452972 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:45.453079 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.453101 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.453176 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.453237 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":101,"threads_started":1}
I20260926 12:28:45.453433 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.449614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:58841 (local address 127.7.99.1:41059)
0926 12:28:45.449749 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.449754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.449898 (+ 144us) server_negotiation.cc:407] Connection header received
0926 12:28:45.450019 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.450021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.450055 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.450109 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:45.450356 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.451080 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452128 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.452747 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.452813 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.452860 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.453218 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.453315 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:45.454051 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e2c016ff1e49d1b9de6fb1b105cbda" candidate_uuid: "668dc2bd05a54150a0aa75964475ca96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb86753662469299f9c94f0f4b3865" is_pre_election: true
I20260926 12:28:45.454030 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e2c016ff1e49d1b9de6fb1b105cbda" candidate_uuid: "668dc2bd05a54150a0aa75964475ca96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d1e74b98154075961687c0206d528b" is_pre_election: true
I20260926 12:28:45.454304 11941 raft_consensus.cc:2454] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668dc2bd05a54150a0aa75964475ca96 in term 0.
I20260926 12:28:45.454315 11808 raft_consensus.cc:2454] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668dc2bd05a54150a0aa75964475ca96 in term 0.
I20260926 12:28:45.454763 12007 leader_election.cc:302] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d1e74b98154075961687c0206d528b, 668dc2bd05a54150a0aa75964475ca96; no voters:
I20260926 12:28:45.454994 12144 raft_consensus.cc:2781] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:45.455070 12144 raft_consensus.cc:482] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:45.455111 12144 raft_consensus.cc:3037] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:28:45.455857 11855 tablet.cc:2369] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:45.456272 12144 raft_consensus.cc:504] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.456640 12144 leader_election.cc:288] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [CANDIDATE]: Term 1 election: Requested vote from peers baeb86753662469299f9c94f0f4b3865 (127.7.99.2:32865), 31d1e74b98154075961687c0206d528b (127.7.99.1:41059)
I20260926 12:28:45.456895 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e2c016ff1e49d1b9de6fb1b105cbda" candidate_uuid: "668dc2bd05a54150a0aa75964475ca96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb86753662469299f9c94f0f4b3865"
I20260926 12:28:45.456984 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e2c016ff1e49d1b9de6fb1b105cbda" candidate_uuid: "668dc2bd05a54150a0aa75964475ca96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d1e74b98154075961687c0206d528b"
I20260926 12:28:45.457023 11941 raft_consensus.cc:3037] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:45.457144 11808 raft_consensus.cc:3037] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:45.458078 11941 raft_consensus.cc:2454] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668dc2bd05a54150a0aa75964475ca96 in term 1.
I20260926 12:28:45.458122 11808 raft_consensus.cc:2454] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668dc2bd05a54150a0aa75964475ca96 in term 1.
I20260926 12:28:45.458369 12009 leader_election.cc:302] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668dc2bd05a54150a0aa75964475ca96, baeb86753662469299f9c94f0f4b3865; no voters:
I20260926 12:28:45.458549 12144 raft_consensus.cc:2781] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:45.458736 12144 raft_consensus.cc:686] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 1 LEADER]: Becoming Leader. State: Replica: 668dc2bd05a54150a0aa75964475ca96, State: Running, Role: LEADER
I20260926 12:28:45.459033 12144 consensus_queue.cc:237] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } }
I20260926 12:28:45.462394 11663 catalog_manager.cc:5495] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668dc2bd05a54150a0aa75964475ca96 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "668dc2bd05a54150a0aa75964475ca96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668dc2bd05a54150a0aa75964475ca96" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:45.483347 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.479799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.7.99.3:33393)
0926 12:28:45.479933 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.479938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.479953 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:28:45.480075 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.480083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.480139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.480275 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:45.480522 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.481032 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.481871 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:45.482089 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:45.482197 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:45.482494 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:45.482901 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:45.483035 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.483110 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.483161 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:45.489130 11808 raft_consensus.cc:1261] T 91e2c016ff1e49d1b9de6fb1b105cbda P 31d1e74b98154075961687c0206d528b [term 1 FOLLOWER]: Refusing update from remote peer 668dc2bd05a54150a0aa75964475ca96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:45.489130 11941 raft_consensus.cc:1261] T 91e2c016ff1e49d1b9de6fb1b105cbda P baeb86753662469299f9c94f0f4b3865 [term 1 FOLLOWER]: Refusing update from remote peer 668dc2bd05a54150a0aa75964475ca96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:45.489802 12147 consensus_queue.cc:1035] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d1e74b98154075961687c0206d528b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:45.490011 12144 consensus_queue.cc:1035] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baeb86753662469299f9c94f0f4b3865" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:45.497634 12152 mvcc.cc:205] Tried to move back new op lower bound from 7333583771596259328 to 7333583771483074560. Current Snapshot: MvccSnapshot[applied={T|T < 7333583771596259328}]
I20260926 12:28:45.507064 11662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:45.507241 11662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:45.509567 11662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 251cd6fa8a07488186e1ece74a2d9de9: Sending DeleteTablet for 3 replicas of tablet 91e2c016ff1e49d1b9de6fb1b105cbda
I20260926 12:28:45.510432 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:49902
I20260926 12:28:45.510447 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:37464
I20260926 12:28:45.510432 11921 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e2c016ff1e49d1b9de6fb1b105cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:58742
I20260926 12:28:45.510931 12160 tablet_replica.cc:330] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: stopping tablet replica
I20260926 12:28:45.511267 12160 raft_consensus.cc:2229] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:45.511462 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11726
I20260926 12:28:45.512352 12160 raft_consensus.cc:2258] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:45.514063 12160 ts_tablet_manager.cc:1905] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:45.517608 12160 ts_tablet_manager.cc:1918] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:45.517710 12160 log.cc:1196] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/91e2c016ff1e49d1b9de6fb1b105cbda
I20260926 12:28:45.518034 12160 ts_tablet_manager.cc:1939] T 91e2c016ff1e49d1b9de6fb1b105cbda P 668dc2bd05a54150a0aa75964475ca96: Deleting consensus metadata
W20260926 12:28:45.518872 11648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:41059 (error 108)
I20260926 12:28:45.519240 11651 catalog_manager.cc:4841] TS 668dc2bd05a54150a0aa75964475ca96 (127.7.99.3:33393): tablet 91e2c016ff1e49d1b9de6fb1b105cbda (table test-table [id=140e939e69db4b03a3b344fcc9a84d0d]) successfully deleted
I20260926 12:28:45.519697 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11859
I20260926 12:28:45.525003 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.524728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:41059 (local address 127.0.0.1:37466)
0926 12:28:45.524857 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.524930 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:28:45.525228 11648 catalog_manager.cc:4584] TS 31d1e74b98154075961687c0206d528b (127.7.99.1:41059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91e2c016ff1e49d1b9de6fb1b105cbda: Network error: Client connection negotiation failed: client connection to 127.7.99.1:41059: connect: Connection refused (error 111)
I20260926 12:28:45.527802 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11992
I20260926 12:28:45.529363 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.529028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:32865 (local address 127.0.0.1:58744)
0926 12:28:45.529210 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.529270 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:32865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 12:28:45.533545 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 11633
2026-09-26T12:28:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2186 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T12:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:45Z Disabled control of system clock
I20260926 12:28:45.563771 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38493
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:28:45.671674 12167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:45.671931 12167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:45.671970 12167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:45.675987 12167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:45.676043 12167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:45.676057 12167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:45.676074 12167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:45.676090 12167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:45.680907 12167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:38493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:45.681952 12167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:45.683318 12167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:45.692226 12175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.692344 12167 server_base.cc:1034] running on GCE node
W20260926 12:28:45.692260 12173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.692211 12172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.693188 12167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:45.694232 12167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:45.695412 12167 hybrid_clock.cc:648] HybridClock initialized: now 1790425725695387 us; error 41 us; skew 500 ppm
I20260926 12:28:45.697204 12167 webserver.cc:460] Webserver started at http://127.7.99.62:44443/ using document root <none> and password file <none>
I20260926 12:28:45.697762 12167 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:45.697850 12167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:45.698081 12167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:45.700065 12167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "685e76da4efe4931b5ae81aef01cc9f9"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "4626902db987c4a0b7416a793400fc0d"
server_key_iv: "48928244695e594b008b58cfd129a356"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.700649 12167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "685e76da4efe4931b5ae81aef01cc9f9"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "4626902db987c4a0b7416a793400fc0d"
server_key_iv: "48928244695e594b008b58cfd129a356"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.704068 12167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:45.706351 12181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.707573 12167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:45.707701 12167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "685e76da4efe4931b5ae81aef01cc9f9"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "4626902db987c4a0b7416a793400fc0d"
server_key_iv: "48928244695e594b008b58cfd129a356"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.707806 12167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:45.712200 12167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:45.712775 12167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:45.712988 12167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:45.719532 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:38493 every 8 connection(s)
I20260926 12:28:45.719530 12167 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:38493
I20260926 12:28:45.720491 12167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:45.723481 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:45.728852 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: Bootstrap starting.
I20260926 12:28:45.729362 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12167
I20260926 12:28:45.729497 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:45.729750 7564 external_mini_cluster.cc:1428] Setting key 6c0cba0793adee8a9d6b40531e2ad627
I20260926 12:28:45.731426 12234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:45.732151 12234 log.cc:824] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:45.734315 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: No bootstrap required, opened a new log
I20260926 12:28:45.734989 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.731061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.7.99.62:38493)
0926 12:28:45.731684 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.731700 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.731733 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:28:45.732431 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.732449 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.732786 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.733045 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:45.733777 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:45.733781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:45.733792 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:45.733800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:45.733852 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:45.733867 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:45.733948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.734079 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.734324 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":134,"threads_started":1}
I20260926 12:28:45.738221 12234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } }
I20260926 12:28:45.738467 12234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:45.738533 12234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685e76da4efe4931b5ae81aef01cc9f9, State: Initialized, Role: FOLLOWER
I20260926 12:28:45.739050 12234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } }
I20260926 12:28:45.739183 12234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:45.739230 12234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:45.739297 12234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:45.740067 12234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } }
I20260926 12:28:45.740408 12234 leader_election.cc:302] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 685e76da4efe4931b5ae81aef01cc9f9; no voters:
I20260926 12:28:45.740666 12234 leader_election.cc:288] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:45.740769 12239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:45.741008 12239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 685e76da4efe4931b5ae81aef01cc9f9, State: Running, Role: LEADER
I20260926 12:28:45.741290 12239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } }
I20260926 12:28:45.741808 12234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:45.742707 12241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 685e76da4efe4931b5ae81aef01cc9f9. Latest consensus state: current_term: 1 leader_uuid: "685e76da4efe4931b5ae81aef01cc9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } } }
I20260926 12:28:45.742817 12241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:45.743216 12248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:45.743176 12240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "685e76da4efe4931b5ae81aef01cc9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e76da4efe4931b5ae81aef01cc9f9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38493 } } }
I20260926 12:28:45.743824 12240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:45.746066 12248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:45.751020 12248 catalog_manager.cc:1347] Generated new cluster ID: f5a1044df6ec4d2e835c5343525ae474
I20260926 12:28:45.751091 12248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:45.766898 12248 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:45.767211 12248 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:45.775235 12248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: Generated new TSK 0
I20260926 12:28:45.775991 12248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:28:45.781424 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--builtin_ntp_servers=127.7.99.20:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:28:45.889215 12258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:45.889492 12258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:45.889575 12258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:45.893543 12258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:45.893616 12258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:45.893705 12258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:45.898480 12258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:45.899529 12258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:45.900903 12258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:45.910224 12266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.910224 12264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:45.910290 12263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:45.910394 12258 server_base.cc:1034] running on GCE node
I20260926 12:28:45.910804 12258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:45.911481 12258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:45.912629 12258 hybrid_clock.cc:648] HybridClock initialized: now 1790425725912605 us; error 35 us; skew 500 ppm
I20260926 12:28:45.914235 12258 webserver.cc:460] Webserver started at http://127.7.99.1:41357/ using document root <none> and password file <none>
I20260926 12:28:45.914775 12258 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:45.914831 12258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:45.915073 12258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:45.917057 12258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "fa3ccd721eda4338bf24d019ae053c2f"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "0c7ecf34d55e54c9c671cdbf9de8cae3"
server_key_iv: "d16cb1c9f762c0d287b66b013c01ff93"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.917594 12258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "fa3ccd721eda4338bf24d019ae053c2f"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "0c7ecf34d55e54c9c671cdbf9de8cae3"
server_key_iv: "d16cb1c9f762c0d287b66b013c01ff93"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.920948 12258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:45.923295 12272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.924458 12258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:45.924574 12258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "fa3ccd721eda4338bf24d019ae053c2f"
format_stamp: "Formatted at 2026-09-26 12:28:45 on dist-test-slave-3jw2"
server_key: "0c7ecf34d55e54c9c671cdbf9de8cae3"
server_key_iv: "d16cb1c9f762c0d287b66b013c01ff93"
server_key_version: "encryptionkey@0"
I20260926 12:28:45.924672 12258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:45.929034 12258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:45.929587 12258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:45.929768 12258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:45.930282 12258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:45.931147 12258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:45.931192 12258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.931257 12258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:45.931288 12258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:45.939440 12258 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:35047
I20260926 12:28:45.939512 12385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:35047 every 8 connection(s)
I20260926 12:28:45.940423 12258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:45.946336 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.942186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:50053 (local address 127.7.99.62:38493)
0926 12:28:45.942321 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:45.942324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:45.943483 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 12:28:45.944865 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:45.944867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:45.944906 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:45.944992 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:45.945970 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:45.945973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:45.945975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:45.945986 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:45.946004 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:45.946013 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:45.946083 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:45.946202 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 12:28:45.946237 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:45.947145 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:45.942616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:38493 (local address 127.7.99.1:50053)
0926 12:28:45.943341 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:45.943380 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:45.944663 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:45.945141 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:45.945147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:45.945249 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:45.945736 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:45.945738 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:45.945775 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:45.945840 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:45.946089 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:45.946095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:45.946099 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:28:45.946222 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 12:28:45.946397 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260926 12:28:45.947319 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12258
I20260926 12:28:45.947444 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:45.947789 7564 external_mini_cluster.cc:1428] Setting key 2654e51eff747ee3ec5be795b7c2e0c9
I20260926 12:28:45.948341 12386 heartbeater.cc:347] Connected to a master server at 127.7.99.62:38493
I20260926 12:28:45.948559 12386 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:45.948951 12386 heartbeater.cc:511] Master 127.7.99.62:38493 requested a full tablet report, sending...
I20260926 12:28:45.949651 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--builtin_ntp_servers=127.7.99.20:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 12:28:45.951397 12198 ts_manager.cc:194] Registered new tserver with Master: fa3ccd721eda4338bf24d019ae053c2f (127.7.99.1:35047)
W20260926 12:28:46.055608 12389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:46.055878 12389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:46.055945 12389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:46.059840 12389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:46.059914 12389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:46.060011 12389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:46.064571 12389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:46.065551 12389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:46.066856 12389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:46.075793 12394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:46.075810 12395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.075949 12389 server_base.cc:1034] running on GCE node
W20260926 12:28:46.075801 12397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.076436 12389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:46.077069 12389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:46.078251 12389 hybrid_clock.cc:648] HybridClock initialized: now 1790425726078240 us; error 31 us; skew 500 ppm
I20260926 12:28:46.079952 12389 webserver.cc:460] Webserver started at http://127.7.99.2:35235/ using document root <none> and password file <none>
I20260926 12:28:46.080549 12389 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:46.080639 12389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:46.080871 12389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:46.082796 12389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d22f8a00474b2258f8cbf7a9915eeeb2"
server_key_iv: "c5f240686f9e98c9535907d7530be230"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.083333 12389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d22f8a00474b2258f8cbf7a9915eeeb2"
server_key_iv: "c5f240686f9e98c9535907d7530be230"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.086808 12389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:28:46.089085 12403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.090164 12389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:46.090291 12389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d22f8a00474b2258f8cbf7a9915eeeb2"
server_key_iv: "c5f240686f9e98c9535907d7530be230"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.090396 12389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:46.094723 12389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:46.095273 12389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:46.095453 12389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:46.095973 12389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:46.096854 12389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:46.096925 12389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.096989 12389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:46.097041 12389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.105252 12389 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:46387
I20260926 12:28:46.105281 12516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:46387 every 8 connection(s)
I20260926 12:28:46.106057 12389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:46.106341 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12389
I20260926 12:28:46.106441 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:46.106739 7564 external_mini_cluster.cc:1428] Setting key f805a02a6d610872d2e1dd83bb74c498
I20260926 12:28:46.108858 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--builtin_ntp_servers=127.7.99.20:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 12:28:46.112661 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.109407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:51851 (local address 127.7.99.62:38493)
0926 12:28:46.109579 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.109586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.109605 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:28:46.110945 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.110950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.110997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.111077 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.112256 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.112259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.112261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.112271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.112287 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.112295 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.112365 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.112523 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.112562 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:28:46.113821 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.108553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:38493 (local address 127.7.99.2:51851)
0926 12:28:46.109229 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.109277 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.110714 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.111254 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.111263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.111378 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.111936 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.111940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.112003 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.112089 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.112380 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.112390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.112399 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.112563 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.112794 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260926 12:28:46.114887 12517 heartbeater.cc:347] Connected to a master server at 127.7.99.62:38493
I20260926 12:28:46.115136 12517 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:46.115599 12517 heartbeater.cc:511] Master 127.7.99.62:38493 requested a full tablet report, sending...
I20260926 12:28:46.116855 12198 ts_manager.cc:194] Registered new tserver with Master: d73fda0dc8f14b1caa6aa2e0a6bc8447 (127.7.99.2:46387)
W20260926 12:28:46.216893 12520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:46.217135 12520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:46.217173 12520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:46.221127 12520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:46.221171 12520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:46.221225 12520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:46.225780 12520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:38493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:46.226723 12520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:46.227977 12520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:46.237289 12525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.237311 12520 server_base.cc:1034] running on GCE node
W20260926 12:28:46.237298 12528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:46.237331 12526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.237908 12520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:46.238511 12520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:46.239702 12520 hybrid_clock.cc:648] HybridClock initialized: now 1790425726239689 us; error 25 us; skew 500 ppm
I20260926 12:28:46.241463 12520 webserver.cc:460] Webserver started at http://127.7.99.3:38989/ using document root <none> and password file <none>
I20260926 12:28:46.242086 12520 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:46.242172 12520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:46.242395 12520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:46.244663 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "77615e44927d43759535e917cb475824"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "be1cbb75597cfe4e29519d8046f9394f"
server_key_iv: "da5fb6f38c67592d79a9ad74eb4d82ec"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.245244 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "77615e44927d43759535e917cb475824"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "be1cbb75597cfe4e29519d8046f9394f"
server_key_iv: "da5fb6f38c67592d79a9ad74eb4d82ec"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.248816 12520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:28:46.251245 12534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.252512 12520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 12:28:46.252645 12520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "77615e44927d43759535e917cb475824"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "be1cbb75597cfe4e29519d8046f9394f"
server_key_iv: "da5fb6f38c67592d79a9ad74eb4d82ec"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.252753 12520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:46.257318 12520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:46.257906 12520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:46.258096 12520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:46.258618 12520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:46.259523 12520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:46.259594 12520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.259663 12520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:46.259716 12520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.268682 12520 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:43641
I20260926 12:28:46.268702 12647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:43641 every 8 connection(s)
I20260926 12:28:46.269608 12520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:46.275524 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12520
I20260926 12:28:46.275635 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.271331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:50565 (local address 127.7.99.62:38493)
0926 12:28:46.271465 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.271470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.272597 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 12:28:46.273865 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.273868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.273914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.274023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.275143 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.275147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.275149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.275160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.275180 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.275190 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.275293 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.275452 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.275504 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:46.275666 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:46.275951 7564 external_mini_cluster.cc:1428] Setting key 9436915f7356d464037bb7aa6cd31365
I20260926 12:28:46.276626 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.271736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:38493 (local address 127.7.99.3:50565)
0926 12:28:46.272456 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.272494 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.273639 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.274180 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.274188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.274272 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.274869 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.274886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.274924 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.274997 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.275276 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.275284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.275289 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.275484 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.275723 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
I20260926 12:28:46.277690 12648 heartbeater.cc:347] Connected to a master server at 127.7.99.62:38493
I20260926 12:28:46.277920 12648 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:46.278327 12648 heartbeater.cc:511] Master 127.7.99.62:38493 requested a full tablet report, sending...
I20260926 12:28:46.279278 12198 ts_manager.cc:194] Registered new tserver with Master: 77615e44927d43759535e917cb475824 (127.7.99.3:43641)
I20260926 12:28:46.290004 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:28:46.293468 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.291918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.7.99.62:38493)
0926 12:28:46.292049 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.292052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.292276 (+ 224us) server_negotiation.cc:407] Connection header received
0926 12:28:46.292387 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.292389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.292424 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.292498 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.293161 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.293164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.293166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.293176 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.293193 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.293201 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.293274 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.293338 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.293368 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:28:46.296126 12198 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:46.299206 12198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:46.310052 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.307336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:46387 (local address 127.0.0.1:44750)
0926 12:28:46.307816 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.307861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.307961 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.308631 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.308638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.308721 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.309262 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.309264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.309284 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.309354 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.309717 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.309725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.309731 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.309840 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.309906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":76,"threads_started":1}
I20260926 12:28:46.310480 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.307470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.7.99.2:46387)
0926 12:28:46.308140 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.308152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.308199 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:28:46.308256 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.308261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.308457 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.308606 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.309505 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.309510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.309523 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.309546 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.309590 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.309621 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.309733 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.310234 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.310308 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":101,"threads_started":1}
I20260926 12:28:46.311268 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.308037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:43641 (local address 127.0.0.1:55868)
0926 12:28:46.309343 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.309353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.309452 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.309841 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.309844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.309859 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.310485 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.310488 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.310499 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.310634 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.311025 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.311027 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.311028 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.311111 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.311145 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":86,"threads_started":1}
I20260926 12:28:46.311367 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.308218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.7.99.3:43641)
0926 12:28:46.308639 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.308650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.309449 (+ 799us) server_negotiation.cc:407] Connection header received
0926 12:28:46.309549 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.309559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.309726 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.309860 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.310786 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.310792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.310804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.310818 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.310858 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.310889 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.310970 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.311128 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.311194 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":102,"threads_started":1}
I20260926 12:28:46.311815 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.307511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35047 (local address 127.0.0.1:50200)
0926 12:28:46.308272 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.308289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.308366 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.310308 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.310311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.310325 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.310788 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.310789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.310799 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.310822 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.311608 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.311610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.311614 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.311680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.311711 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":87,"threads_started":1}
I20260926 12:28:46.312232 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.308732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.7.99.1:35047)
0926 12:28:46.309374 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.309386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.309410 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:28:46.309466 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.309471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.309650 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.309774 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.311032 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.311036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.311046 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.311055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.311087 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.311107 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.311952 (+ 845us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.312001 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.312060 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":157,"threads_started":1}
I20260926 12:28:46.313499 12451 tablet_service.cc:1467] Processing CreateTablet for tablet c7cc17c1167a4e598476bcd733db5108 (DEFAULT_TABLE table=test-table [id=f8b26d41e83e44d79c68f075b9f69432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:46.314438 12451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cc17c1167a4e598476bcd733db5108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:46.314560 12582 tablet_service.cc:1467] Processing CreateTablet for tablet c7cc17c1167a4e598476bcd733db5108 (DEFAULT_TABLE table=test-table [id=f8b26d41e83e44d79c68f075b9f69432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:46.314546 12320 tablet_service.cc:1467] Processing CreateTablet for tablet c7cc17c1167a4e598476bcd733db5108 (DEFAULT_TABLE table=test-table [id=f8b26d41e83e44d79c68f075b9f69432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:46.315536 12320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cc17c1167a4e598476bcd733db5108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:46.315546 12582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cc17c1167a4e598476bcd733db5108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:46.320703 12665 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Bootstrap starting.
I20260926 12:28:46.322336 12666 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Bootstrap starting.
I20260926 12:28:46.322942 12665 tablet_bootstrap.cc:654] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:46.323228 12667 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Bootstrap starting.
I20260926 12:28:46.323909 12665 log.cc:824] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:46.324458 12666 tablet_bootstrap.cc:654] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:46.325105 12667 tablet_bootstrap.cc:654] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:46.325317 12666 log.cc:824] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:46.325870 12667 log.cc:824] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:46.326468 12665 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: No bootstrap required, opened a new log
I20260926 12:28:46.326684 12665 ts_tablet_manager.cc:1397] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:28:46.327210 12666 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: No bootstrap required, opened a new log
I20260926 12:28:46.327385 12666 ts_tablet_manager.cc:1397] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:46.327677 12667 tablet_bootstrap.cc:492] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: No bootstrap required, opened a new log
I20260926 12:28:46.327841 12667 ts_tablet_manager.cc:1397] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:28:46.330593 12665 raft_consensus.cc:348] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.330859 12665 raft_consensus.cc:374] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:46.330915 12665 raft_consensus.cc:729] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73fda0dc8f14b1caa6aa2e0a6bc8447, State: Initialized, Role: FOLLOWER
I20260926 12:28:46.331382 12665 consensus_queue.cc:260] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.331839 12666 raft_consensus.cc:348] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.332020 12517 heartbeater.cc:503] Master 127.7.99.62:38493 was elected leader, sending a full tablet report...
I20260926 12:28:46.332083 12666 raft_consensus.cc:374] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:46.332132 12666 raft_consensus.cc:729] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77615e44927d43759535e917cb475824, State: Initialized, Role: FOLLOWER
I20260926 12:28:46.332139 12667 raft_consensus.cc:348] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.332366 12667 raft_consensus.cc:374] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:46.332403 12665 ts_tablet_manager.cc:1428] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:28:46.332418 12667 raft_consensus.cc:729] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3ccd721eda4338bf24d019ae053c2f, State: Initialized, Role: FOLLOWER
I20260926 12:28:46.332607 12666 consensus_queue.cc:260] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.332805 12667 consensus_queue.cc:260] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.333202 12648 heartbeater.cc:503] Master 127.7.99.62:38493 was elected leader, sending a full tablet report...
I20260926 12:28:46.333666 12667 ts_tablet_manager.cc:1428] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:28:46.333698 12666 ts_tablet_manager.cc:1428] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:28:46.333963 12386 heartbeater.cc:503] Master 127.7.99.62:38493 was elected leader, sending a full tablet report...
W20260926 12:28:46.357146 12518 tablet.cc:2369] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:46.441413 12673 raft_consensus.cc:482] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 12:28:46.441534 12387 tablet.cc:2369] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:46.441674 12673 raft_consensus.cc:504] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.442844 12673 leader_election.cc:288] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73fda0dc8f14b1caa6aa2e0a6bc8447 (127.7.99.2:46387), 77615e44927d43759535e917cb475824 (127.7.99.3:43641)
I20260926 12:28:46.445223 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.443416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:50473 (local address 127.7.99.3:43641)
0926 12:28:46.443588 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.443593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.443850 (+ 257us) server_negotiation.cc:407] Connection header received
0926 12:28:46.443962 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.443967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.444012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.444107 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.444829 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.444832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.444835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.444848 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.444864 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.444872 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.444974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.445046 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.445083 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:28:46.445261 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.443372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:55445 (local address 127.7.99.2:46387)
0926 12:28:46.443547 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.443555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.443576 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:28:46.443629 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.443632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.443673 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.443772 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.444700 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.444706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.444710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.444725 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.444758 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.444769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.444941 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.445067 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.445110 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:28:46.445263 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.443116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:46387 (local address 127.7.99.1:55445)
0926 12:28:46.443332 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.443351 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.443459 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.443850 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.443854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.443873 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.444550 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.444551 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.444561 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.444583 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.444929 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.444933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.444937 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.445066 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.445110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 12:28:46.445267 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.443117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:43641 (local address 127.7.99.1:50473)
0926 12:28:46.443742 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.443759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.443858 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.444177 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.444181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.444200 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:46.444710 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:28:46.444711 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:46.444717 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:28:46.444729 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:46.444960 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:46.444962 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:46.444965 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:28:46.445044 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:28:46.445083 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":170,"thread_start_us":118,"threads_started":1}
I20260926 12:28:46.445874 12471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cc17c1167a4e598476bcd733db5108" candidate_uuid: "fa3ccd721eda4338bf24d019ae053c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" is_pre_election: true
I20260926 12:28:46.445964 12602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cc17c1167a4e598476bcd733db5108" candidate_uuid: "fa3ccd721eda4338bf24d019ae053c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77615e44927d43759535e917cb475824" is_pre_election: true
I20260926 12:28:46.446187 12471 raft_consensus.cc:2454] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa3ccd721eda4338bf24d019ae053c2f in term 0.
I20260926 12:28:46.446312 12602 raft_consensus.cc:2454] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa3ccd721eda4338bf24d019ae053c2f in term 0.
I20260926 12:28:46.446700 12274 leader_election.cc:302] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d73fda0dc8f14b1caa6aa2e0a6bc8447, fa3ccd721eda4338bf24d019ae053c2f; no voters:
I20260926 12:28:46.446967 12673 raft_consensus.cc:2781] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:46.447067 12673 raft_consensus.cc:482] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:46.447132 12673 raft_consensus.cc:3037] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:46.448401 12673 raft_consensus.cc:504] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.448822 12673 leader_election.cc:288] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [CANDIDATE]: Term 1 election: Requested vote from peers d73fda0dc8f14b1caa6aa2e0a6bc8447 (127.7.99.2:46387), 77615e44927d43759535e917cb475824 (127.7.99.3:43641)
I20260926 12:28:46.449158 12471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cc17c1167a4e598476bcd733db5108" candidate_uuid: "fa3ccd721eda4338bf24d019ae053c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447"
I20260926 12:28:46.449203 12602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cc17c1167a4e598476bcd733db5108" candidate_uuid: "fa3ccd721eda4338bf24d019ae053c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77615e44927d43759535e917cb475824"
I20260926 12:28:46.449308 12471 raft_consensus.cc:3037] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:46.449329 12602 raft_consensus.cc:3037] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:46.450739 12602 raft_consensus.cc:2454] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa3ccd721eda4338bf24d019ae053c2f in term 1.
I20260926 12:28:46.450739 12471 raft_consensus.cc:2454] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa3ccd721eda4338bf24d019ae053c2f in term 1.
I20260926 12:28:46.451098 12275 leader_election.cc:302] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77615e44927d43759535e917cb475824, fa3ccd721eda4338bf24d019ae053c2f; no voters:
I20260926 12:28:46.451310 12673 raft_consensus.cc:2781] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:46.451561 12673 raft_consensus.cc:686] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [term 1 LEADER]: Becoming Leader. State: Replica: fa3ccd721eda4338bf24d019ae053c2f, State: Running, Role: LEADER
I20260926 12:28:46.451879 12673 consensus_queue.cc:237] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } }
I20260926 12:28:46.454912 12198 catalog_manager.cc:5495] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fa3ccd721eda4338bf24d019ae053c2f (127.7.99.1). New cstate: current_term: 1 leader_uuid: "fa3ccd721eda4338bf24d019ae053c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa3ccd721eda4338bf24d019ae053c2f" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:46.463985 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.462651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.7.99.1:35047)
0926 12:28:46.462778 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.462782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.462797 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:28:46.462896 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.462899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.462958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.463044 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.463670 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.463673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.463676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.463687 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.463707 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.463716 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.463800 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.463853 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.463888 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:46.469549 12471 raft_consensus.cc:1261] T c7cc17c1167a4e598476bcd733db5108 P d73fda0dc8f14b1caa6aa2e0a6bc8447 [term 1 FOLLOWER]: Refusing update from remote peer fa3ccd721eda4338bf24d019ae053c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:46.469559 12602 raft_consensus.cc:1261] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 1 FOLLOWER]: Refusing update from remote peer fa3ccd721eda4338bf24d019ae053c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:46.470252 12673 consensus_queue.cc:1035] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73fda0dc8f14b1caa6aa2e0a6bc8447" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:46.470429 12675 consensus_queue.cc:1035] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77615e44927d43759535e917cb475824" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:46.487228 12198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:46.487401 12198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:46.489661 12198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 685e76da4efe4931b5ae81aef01cc9f9: Sending DeleteTablet for 3 replicas of tablet c7cc17c1167a4e598476bcd733db5108
I20260926 12:28:46.490365 12451 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cc17c1167a4e598476bcd733db5108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:46 UTC) from {username='slave'} at 127.0.0.1:44750
I20260926 12:28:46.490504 12582 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cc17c1167a4e598476bcd733db5108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:46 UTC) from {username='slave'} at 127.0.0.1:55868
I20260926 12:28:46.490535 12320 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cc17c1167a4e598476bcd733db5108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:46 UTC) from {username='slave'} at 127.0.0.1:50200
I20260926 12:28:46.491016 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12258
I20260926 12:28:46.491128 12690 tablet_replica.cc:330] T c7cc17c1167a4e598476bcd733db5108 P fa3ccd721eda4338bf24d019ae053c2f: stopping tablet replica
I20260926 12:28:46.491659 12689 tablet_replica.cc:330] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: stopping tablet replica
I20260926 12:28:46.491914 12689 raft_consensus.cc:2229] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:46.492234 12689 raft_consensus.cc:2258] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:28:46.496639 12184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:35047 (error 108)
I20260926 12:28:46.496793 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12389
I20260926 12:28:46.498238 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.497791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35047 (local address 127.0.0.1:50206)
0926 12:28:46.498065 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.498141 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260926 12:28:46.498476 12184 catalog_manager.cc:4584] TS fa3ccd721eda4338bf24d019ae053c2f (127.7.99.1:35047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7cc17c1167a4e598476bcd733db5108: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35047: connect: Connection refused (error 111)
I20260926 12:28:46.499096 12689 ts_tablet_manager.cc:1905] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:46.501385 12689 ts_tablet_manager.cc:1918] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:46.501488 12689 log.cc:1196] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/c7cc17c1167a4e598476bcd733db5108
I20260926 12:28:46.501767 12689 ts_tablet_manager.cc:1939] T c7cc17c1167a4e598476bcd733db5108 P 77615e44927d43759535e917cb475824: Deleting consensus metadata
I20260926 12:28:46.502811 12184 catalog_manager.cc:4841] TS 77615e44927d43759535e917cb475824 (127.7.99.3:43641): tablet c7cc17c1167a4e598476bcd733db5108 (table test-table [id=f8b26d41e83e44d79c68f075b9f69432]) successfully deleted
I20260926 12:28:46.502907 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12520
I20260926 12:28:46.504832 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.503098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:46387 (local address 127.0.0.1:44762)
0926 12:28:46.504669 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.504735 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1466}
I20260926 12:28:46.508611 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12167
2026-09-26T12:28:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (974 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 12:28:46.530033 7564 mini_oidc.cc:152] Starting JWKS server
I20260926 12:28:46.531589 7564 webserver.cc:460] Webserver started at https://127.0.0.1:45295/ using document root <none> and password file <none>
I20260926 12:28:46.531769 7564 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:28:46.532032 7564 webserver.cc:460] Webserver started at http://127.0.0.1:46845/ using document root <none> and password file <none>
2026-09-26T12:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:46Z Disabled control of system clock
I20260926 12:28:46.540522 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45293
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:45293
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/testchainca.cert with env {}
W20260926 12:28:46.658452 12697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:46.658689 12697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:46.658756 12697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:46.662640 12697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:46.662688 12697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:46.662701 12697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:46.662716 12697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:46.662729 12697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:28:46.662743 12697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:28:46.662755 12697 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45295/jwks.json
I20260926 12:28:46.668089 12697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:45293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:46.669109 12697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:46.670145 12697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:46.678483 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:46.678475 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.678583 12697 server_base.cc:1034] running on GCE node
W20260926 12:28:46.678484 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.679181 12697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:46.680265 12697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:46.681447 12697 hybrid_clock.cc:648] HybridClock initialized: now 1790425726681434 us; error 28 us; skew 500 ppm
I20260926 12:28:46.683275 12697 webserver.cc:460] Webserver started at http://127.7.99.62:38671/ using document root <none> and password file <none>
I20260926 12:28:46.683862 12697 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:46.683952 12697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:46.684230 12697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:46.686187 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "3fb9ad10627a4b539b96bb2484bf48b3"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "acf448ad1a537f4c5a4d305263f9ab43"
server_key_iv: "52746f20a520b258113f5df285a1bae5"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.686734 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "3fb9ad10627a4b539b96bb2484bf48b3"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "acf448ad1a537f4c5a4d305263f9ab43"
server_key_iv: "52746f20a520b258113f5df285a1bae5"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.690219 12697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 12:28:46.692510 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:46.693655 12697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:46.693790 12697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "3fb9ad10627a4b539b96bb2484bf48b3"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "acf448ad1a537f4c5a4d305263f9ab43"
server_key_iv: "52746f20a520b258113f5df285a1bae5"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.693897 12697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:46.724414 12697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:46.725077 12697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:46.725261 12697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:46.731814 12697 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:45293
I20260926 12:28:46.731812 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:45293 every 8 connection(s)
I20260926 12:28:46.732848 12697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:46.735477 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:46.736680 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12697
I20260926 12:28:46.736788 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:46.737097 7564 external_mini_cluster.cc:1428] Setting key 86de62873079556670671a7849d38169
I20260926 12:28:46.741537 12764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: Bootstrap starting.
I20260926 12:28:46.743536 12764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:46.744248 12764 log.cc:824] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:46.745740 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.738354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.7.99.62:45293)
0926 12:28:46.738948 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.738961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.738994 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:28:46.739626 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.739641 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.740025 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.740406 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:46.740827 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.741808 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.742651 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.743093 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.743260 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:46.744092 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:46.744103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:46.744115 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:46.744576 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:46.744626 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:46.744645 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:46.744748 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:46.744850 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:28:46.745100 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260926 12:28:46.746055 12764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: No bootstrap required, opened a new log
I20260926 12:28:46.748436 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:28:46.750183 12764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } }
I20260926 12:28:46.750401 12764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:46.750453 12764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb9ad10627a4b539b96bb2484bf48b3, State: Initialized, Role: FOLLOWER
I20260926 12:28:46.750842 12764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } }
I20260926 12:28:46.750990 12764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:46.751077 12764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:46.751176 12764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:46.752223 12764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } }
I20260926 12:28:46.752525 12764 leader_election.cc:302] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fb9ad10627a4b539b96bb2484bf48b3; no voters:
I20260926 12:28:46.752745 12764 leader_election.cc:288] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:46.752871 12769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:46.753131 12769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb9ad10627a4b539b96bb2484bf48b3, State: Running, Role: LEADER
I20260926 12:28:46.753412 12769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } }
I20260926 12:28:46.753870 12764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:46.755264 12771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fb9ad10627a4b539b96bb2484bf48b3. Latest consensus state: current_term: 1 leader_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } } }
I20260926 12:28:46.755404 12771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:46.755246 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb9ad10627a4b539b96bb2484bf48b3" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45293 } } }
I20260926 12:28:46.755625 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3 [sys.catalog]: This master's current role is: LEADER
W20260926 12:28:46.758355 12785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:28:46.758416 12785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:28:46.758491 12780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:46.759126 12780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:46.764000 12780 catalog_manager.cc:1347] Generated new cluster ID: 25d5ec666c85456b993fff071eac7ca6
I20260926 12:28:46.764055 12780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:46.776037 12780 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:46.776991 12780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:46.787243 12780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fb9ad10627a4b539b96bb2484bf48b3: Generated new TSK 0
I20260926 12:28:46.787973 12780 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 12:28:46.816138 12794 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 12:28:46.816293 12794 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:28:46.816785 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:46.800421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.7.99.62:45293)
0926 12:28:46.800546 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:46.800551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:46.800737 (+ 186us) server_negotiation.cc:407] Connection header received
0926 12:28:46.800901 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:46.800904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:46.800958 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:46.801070 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:28:46.801275 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.802095 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.802857 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.803183 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.803275 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:46.803521 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:28:46.816418 (+ 12897us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:28:46.816647 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 12:28:46.816857 12767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 12:28:46.817826 12793 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:46.800314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:45293 (local address 127.0.0.1:51832)
0926 12:28:46.800656 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:46.800668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:46.800764 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:46.801076 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:46.801079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:46.801085 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:28:46.801180 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:46.801185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.802186 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.802188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:46.802759 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:46.802765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:46.803270 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:46.803282 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:46.803339 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:46.803372 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:28:46.816597 (+ 13225us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:28:46.816757 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.99.62:45293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.99.62:45293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 12:28:46.818552 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12697
2026-09-26T12:28:46Z chronyd exiting
I20260926 12:28:46.855389 7564 test_util.cc:175] -----------------------------------------------
I20260926 12:28:46.855464 7564 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425700812969-7564-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 12:28:46.856557 7564 mini_oidc.cc:152] Starting JWKS server
I20260926 12:28:46.857383 7564 webserver.cc:460] Webserver started at https://127.0.0.1:43801/ using document root <none> and password file <none>
I20260926 12:28:46.857498 7564 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:28:46.857777 7564 webserver.cc:460] Webserver started at http://127.0.0.1:37279/ using document root <none> and password file <none>
2026-09-26T12:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:46Z Disabled control of system clock
I20260926 12:28:46.866354 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38277
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:38277
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/testchainca.cert with env {}
W20260926 12:28:46.971488 12800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:46.971736 12800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:46.971773 12800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:46.975621 12800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:46.975673 12800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:46.975689 12800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:46.975705 12800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:46.975720 12800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:28:46.975735 12800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:28:46.975749 12800 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43801/jwks.json
I20260926 12:28:46.980868 12800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:38277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:46.981782 12800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:46.982695 12800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:46.990898 12805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:46.990957 12806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.991142 12800 server_base.cc:1034] running on GCE node
W20260926 12:28:46.990899 12808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:46.991794 12800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:46.992825 12800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:46.993978 12800 hybrid_clock.cc:648] HybridClock initialized: now 1790425726993962 us; error 29 us; skew 500 ppm
I20260926 12:28:46.995641 12800 webserver.cc:460] Webserver started at http://127.7.99.62:45079/ using document root <none> and password file <none>
I20260926 12:28:46.996146 12800 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:46.996284 12800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:46.996495 12800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:46.998350 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "ec9db895352d49f992721e2e4f2c1ebe"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d5a993dd56b2dabf9db5bf521b181d2d"
server_key_iv: "258467330d70c9bea2f358e22459ccd3"
server_key_version: "encryptionkey@0"
I20260926 12:28:46.998823 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "ec9db895352d49f992721e2e4f2c1ebe"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d5a993dd56b2dabf9db5bf521b181d2d"
server_key_iv: "258467330d70c9bea2f358e22459ccd3"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.002162 12800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:28:47.004344 12814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:47.005460 12800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:47.005553 12800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "ec9db895352d49f992721e2e4f2c1ebe"
format_stamp: "Formatted at 2026-09-26 12:28:46 on dist-test-slave-3jw2"
server_key: "d5a993dd56b2dabf9db5bf521b181d2d"
server_key_iv: "258467330d70c9bea2f358e22459ccd3"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.005633 12800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:47.034621 12800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:47.035239 12800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:47.035380 12800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:47.042047 12800 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:38277
I20260926 12:28:47.042053 12866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:38277 every 8 connection(s)
I20260926 12:28:47.042917 12800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:47.045817 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:47.050957 12867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: Bootstrap starting.
I20260926 12:28:47.051906 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12800
I20260926 12:28:47.051997 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:47.052345 7564 external_mini_cluster.cc:1428] Setting key ff83b9f77c98f095b79f957831323707
I20260926 12:28:47.053599 12867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:47.054279 12867 log.cc:824] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:47.056084 12867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: No bootstrap required, opened a new log
I20260926 12:28:47.059455 12867 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } }
I20260926 12:28:47.059650 12867 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:47.059711 12867 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9db895352d49f992721e2e4f2c1ebe, State: Initialized, Role: FOLLOWER
I20260926 12:28:47.060153 12867 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } }
I20260926 12:28:47.060360 12867 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:47.060429 12867 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:47.060529 12867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:47.061128 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.053357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.7.99.62:38277)
0926 12:28:47.054000 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.054013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.054051 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:28:47.054733 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.054755 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.055165 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.055489 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:47.055904 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.056909 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.057951 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.058433 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.058603 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.059388 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:47.059393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:47.059404 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:47.059839 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:47.059891 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:47.059911 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:47.060012 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:47.060119 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 12:28:47.060411 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260926 12:28:47.061299 12867 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } }
I20260926 12:28:47.061596 12867 leader_election.cc:302] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9db895352d49f992721e2e4f2c1ebe; no voters:
I20260926 12:28:47.061878 12867 leader_election.cc:288] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:47.061939 12872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:47.062876 12872 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [term 1 LEADER]: Becoming Leader. State: Replica: ec9db895352d49f992721e2e4f2c1ebe, State: Running, Role: LEADER
I20260926 12:28:47.063071 12867 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:47.063146 12872 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } }
I20260926 12:28:47.063880 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:28:47.065388 12873 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9db895352d49f992721e2e4f2c1ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } } }
I20260926 12:28:47.065577 12874 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9db895352d49f992721e2e4f2c1ebe. Latest consensus state: current_term: 1 leader_uuid: "ec9db895352d49f992721e2e4f2c1ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9db895352d49f992721e2e4f2c1ebe" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 38277 } } }
I20260926 12:28:47.066151 12873 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:47.066156 12874 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe [sys.catalog]: This master's current role is: LEADER
W20260926 12:28:47.067857 12886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:28:47.067929 12886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:28:47.068032 12888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:47.068682 12888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:47.073616 12888 catalog_manager.cc:1347] Generated new cluster ID: 40dc4b68c83e41b9ad4b259348f626d4
I20260926 12:28:47.073674 12888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:47.091472 12888 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:47.092624 12888 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:47.109436 12888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9db895352d49f992721e2e4f2c1ebe: Generated new TSK 0
I20260926 12:28:47.110188 12888 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 12:28:47.137130 12897 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 12:28:47.137209 12897 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:28:47.137771 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.122068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.7.99.62:38277)
0926 12:28:47.122175 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.122180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.122456 (+ 276us) server_negotiation.cc:407] Connection header received
0926 12:28:47.122648 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.122651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.122734 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.122896 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:28:47.123131 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.123781 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.124979 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.125185 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.125258 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.125564 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:28:47.137493 (+ 11929us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:28:47.137672 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 12:28:47.137847 12870 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 12:28:47.137972 12896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:47.121965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:38277 (local address 127.0.0.1:40808)
0926 12:28:47.122376 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:47.122391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:47.122498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:47.122890 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:47.122894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:47.122901 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:28:47.123018 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.123024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.123892 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.123896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.124866 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.124876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.125275 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.125293 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.125386 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.125418 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:28:47.137648 (+ 12230us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:28:47.137809 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.99.62:38277: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
I20260926 12:28:47.138548 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12800
2026-09-26T12:28:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 12:28:47.177740 7564 mini_oidc.cc:152] Starting JWKS server
I20260926 12:28:47.178601 7564 webserver.cc:460] Webserver started at https://127.0.0.1:38441/ using document root <none> and password file <none>
I20260926 12:28:47.178730 7564 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:28:47.178995 7564 webserver.cc:460] Webserver started at http://127.0.0.1:37685/ using document root <none> and password file <none>
2026-09-26T12:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:47Z Disabled control of system clock
I20260926 12:28:47.187544 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40743
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:40743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38441/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:28:47.293066 12903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:47.293308 12903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:47.293344 12903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:47.297147 12903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:47.297190 12903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:47.297204 12903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:47.297221 12903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:47.297235 12903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:28:47.297251 12903 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:28:47.297264 12903 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38441/jwks.json
I20260926 12:28:47.301741 12903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:40743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38441/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:47.302750 12903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:47.304056 12903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:47.311975 12908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:47.312013 12909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:47.312009 12911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:47.312227 12903 server_base.cc:1034] running on GCE node
I20260926 12:28:47.312858 12903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:47.313897 12903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:47.315104 12903 hybrid_clock.cc:648] HybridClock initialized: now 1790425727315087 us; error 29 us; skew 500 ppm
I20260926 12:28:47.316851 12903 webserver.cc:460] Webserver started at http://127.7.99.62:40239/ using document root <none> and password file <none>
I20260926 12:28:47.317405 12903 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:47.317495 12903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:47.317720 12903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:47.319640 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "14766bfb4fd24f87a54077ea5c29b384"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "6adba8cb2eb74c40fb6964e4c6c52e65"
server_key_iv: "3068a2a9109b93c3db17057411804c4c"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.320215 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "14766bfb4fd24f87a54077ea5c29b384"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "6adba8cb2eb74c40fb6964e4c6c52e65"
server_key_iv: "3068a2a9109b93c3db17057411804c4c"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.323633 12903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:28:47.325941 12917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:47.327106 12903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:47.327247 12903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "14766bfb4fd24f87a54077ea5c29b384"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "6adba8cb2eb74c40fb6964e4c6c52e65"
server_key_iv: "3068a2a9109b93c3db17057411804c4c"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.327358 12903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:47.338938 12903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:47.339586 12903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:47.339778 12903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:47.346237 12903 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:40743
I20260926 12:28:47.346271 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:40743 every 8 connection(s)
I20260926 12:28:47.347148 12903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:47.350270 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:47.352721 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 12903
I20260926 12:28:47.352955 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:47.353272 7564 external_mini_cluster.cc:1428] Setting key 40f182e1049d666ad1434eceecef044f
I20260926 12:28:47.356621 12970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384: Bootstrap starting.
I20260926 12:28:47.358747 12970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:47.359498 12970 log.cc:824] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:47.361655 12970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384: No bootstrap required, opened a new log
I20260926 12:28:47.362679 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.354667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.7.99.62:40743)
0926 12:28:47.355312 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.355326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.355361 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:28:47.355995 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.356011 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.356470 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.356799 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:47.357238 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.358178 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.359137 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.359592 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.359758 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.360712 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:47.360718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:47.360733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:47.361205 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:47.361275 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:47.361300 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:47.361402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:47.361529 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 12:28:47.361790 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260926 12:28:47.365586 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:28:47.365857 12970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } }
I20260926 12:28:47.366135 12970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:47.366192 12970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14766bfb4fd24f87a54077ea5c29b384, State: Initialized, Role: FOLLOWER
I20260926 12:28:47.366613 12970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } }
I20260926 12:28:47.366776 12970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:47.366842 12970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:47.366937 12970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:47.367836 12970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } }
I20260926 12:28:47.368119 12970 leader_election.cc:302] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14766bfb4fd24f87a54077ea5c29b384; no voters:
I20260926 12:28:47.368426 12970 leader_election.cc:288] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:47.368542 12975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:47.368788 12975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [term 1 LEADER]: Becoming Leader. State: Replica: 14766bfb4fd24f87a54077ea5c29b384, State: Running, Role: LEADER
I20260926 12:28:47.369089 12975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } }
I20260926 12:28:47.369657 12970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:47.370710 12976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14766bfb4fd24f87a54077ea5c29b384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } } }
I20260926 12:28:47.370828 12976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:47.371109 12977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14766bfb4fd24f87a54077ea5c29b384. Latest consensus state: current_term: 1 leader_uuid: "14766bfb4fd24f87a54077ea5c29b384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14766bfb4fd24f87a54077ea5c29b384" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40743 } } }
I20260926 12:28:47.371213 12977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:47.371301 12984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:47.374370 12984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:47.379276 12984 catalog_manager.cc:1347] Generated new cluster ID: fe5b434e309b4d11822f76086ecd580a
I20260926 12:28:47.379343 12984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:47.403707 12984 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:47.404910 12984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:47.415176 12984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14766bfb4fd24f87a54077ea5c29b384: Generated new TSK 0
I20260926 12:28:47.415865 12984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:28:47.433250 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.428125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.7.99.62:40743)
0926 12:28:47.428273 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.428278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.428478 (+ 200us) server_negotiation.cc:407] Connection header received
0926 12:28:47.428638 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.428640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.428685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.428772 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:28:47.429010 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.429519 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.430802 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.431014 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.431090 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.431427 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:28:47.432906 (+ 1479us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:28:47.433118 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 12:28:47.433313 12973 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 12:28:47.435113 12999 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:47.428069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:40743 (local address 127.0.0.1:38502)
0926 12:28:47.428385 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:47.428400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:47.428519 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:47.428794 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:47.428797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:47.428804 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:28:47.428914 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.428920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.429630 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.429634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.430667 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.430697 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.431103 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.431124 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.431211 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.431237 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:28:47.433082 (+ 1845us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:28:47.434940 (+ 1858us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.99.62:40743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1658,"thread_start_us":135,"threads_started":1}
I20260926 12:28:47.435653 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 12903
E20260926 12:28:47.446161 13000 webserver.cc:533] Webserver: sslize failed:
E20260926 12:28:47.446225 13000 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T12:28:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T12:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:47Z Disabled control of system clock
I20260926 12:28:47.483996 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:39103
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:39103
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/testchainca.cert with env {}
W20260926 12:28:47.589514 13004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:47.589756 13004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:47.589792 13004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:47.593613 13004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:47.593657 13004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:47.593670 13004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:47.593685 13004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:47.593699 13004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:28:47.593714 13004 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:28:47.593727 13004 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 12:28:47.598757 13004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:39103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:39103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:47.599611 13004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:47.600481 13004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:47.608518 13009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:47.608526 13010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:47.608561 13004 server_base.cc:1034] running on GCE node
W20260926 12:28:47.608516 13012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:47.609179 13004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:47.610236 13004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:47.611415 13004 hybrid_clock.cc:648] HybridClock initialized: now 1790425727611373 us; error 56 us; skew 500 ppm
I20260926 12:28:47.613168 13004 webserver.cc:460] Webserver started at http://127.7.99.62:35517/ using document root <none> and password file <none>
I20260926 12:28:47.613708 13004 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:47.613796 13004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:47.614009 13004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:47.615839 13004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "2b3ac2d6a8f44872a87378ebc47b402d"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "e13827147fe118e4e035f22cf6f5657a"
server_key_iv: "d7b17daf50a56056639721d048ed4eeb"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.616402 13004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "2b3ac2d6a8f44872a87378ebc47b402d"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "e13827147fe118e4e035f22cf6f5657a"
server_key_iv: "d7b17daf50a56056639721d048ed4eeb"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.619771 13004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 12:28:47.621950 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:47.622985 13004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:47.623121 13004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "2b3ac2d6a8f44872a87378ebc47b402d"
format_stamp: "Formatted at 2026-09-26 12:28:47 on dist-test-slave-3jw2"
server_key: "e13827147fe118e4e035f22cf6f5657a"
server_key_iv: "d7b17daf50a56056639721d048ed4eeb"
server_key_version: "encryptionkey@0"
I20260926 12:28:47.623224 13004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:47.635553 13004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:47.636124 13004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:47.636343 13004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:47.642562 13004 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:39103
I20260926 12:28:47.642603 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:39103 every 8 connection(s)
I20260926 12:28:47.643397 13004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:47.646236 13071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:47.649227 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13004
I20260926 12:28:47.649327 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:47.649621 7564 external_mini_cluster.cc:1428] Setting key cb120d3e55cb32ceca1fd806dcdf4f50
I20260926 12:28:47.651629 13071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d: Bootstrap starting.
I20260926 12:28:47.653817 13071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:47.654506 13071 log.cc:824] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:47.656251 13071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d: No bootstrap required, opened a new log
I20260926 12:28:47.658242 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.650911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.7.99.62:39103)
0926 12:28:47.651481 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.651494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.651526 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:28:47.652191 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.652222 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.652582 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.652883 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:47.653334 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.654427 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.655288 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.655723 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.655913 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.656718 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:47.656724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:47.656744 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:28:47.657113 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:47.657163 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:28:47.657181 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:47.657284 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:47.657404 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 12:28:47.657634 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260926 12:28:47.659686 13071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } }
I20260926 12:28:47.659981 13071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:47.660051 13071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3ac2d6a8f44872a87378ebc47b402d, State: Initialized, Role: FOLLOWER
I20260926 12:28:47.660552 13071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } }
I20260926 12:28:47.660701 13071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:47.660764 13071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:47.660867 13071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:47.660902 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:28:47.661669 13071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } }
I20260926 12:28:47.662027 13071 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3ac2d6a8f44872a87378ebc47b402d; no voters:
I20260926 12:28:47.662320 13071 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:47.662413 13076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:47.662621 13076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [term 1 LEADER]: Becoming Leader. State: Replica: 2b3ac2d6a8f44872a87378ebc47b402d, State: Running, Role: LEADER
I20260926 12:28:47.662887 13076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } }
I20260926 12:28:47.663393 13071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:47.664353 13078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3ac2d6a8f44872a87378ebc47b402d. Latest consensus state: current_term: 1 leader_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } } }
I20260926 12:28:47.664476 13078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:47.664748 13077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ac2d6a8f44872a87378ebc47b402d" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39103 } } }
I20260926 12:28:47.664819 13085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:47.664844 13077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:47.667603 13085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:47.673424 13085 catalog_manager.cc:1347] Generated new cluster ID: 59d278c0437d47279e7af1e622711676
I20260926 12:28:47.673516 13085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:47.688541 13085 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:47.689416 13085 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:47.699287 13085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3ac2d6a8f44872a87378ebc47b402d: Generated new TSK 0
I20260926 12:28:47.699924 13085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:28:47.743052 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:47.713010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.7.99.62:39103)
0926 12:28:47.713139 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:47.713144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:47.713324 (+ 180us) server_negotiation.cc:407] Connection header received
0926 12:28:47.713486 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:47.713488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:47.713530 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:47.713611 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:28:47.713845 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.714383 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.715515 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.715712 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.715780 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.716103 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:28:47.742699 (+ 26596us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:28:47.742930 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50570: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:28:47.743145 13074 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50570: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 12:28:47.743528 13100 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:47.712951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:39103 (local address 127.0.0.1:50570)
0926 12:28:47.713236 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:47.713251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:47.713359 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:47.713637 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:47.713640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:47.713648 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:28:47.713749 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.713755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.714482 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.714486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.715410 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:47.715421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:47.715796 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:47.715817 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:47.715933 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:47.715959 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:28:47.742900 (+ 26941us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:28:47.743088 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.99.62:39103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260926 12:28:47.744118 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13004
2026-09-26T12:28:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (287 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:48 dist-test-slave-3jw2 krb5kdc[13112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:48 dist-test-slave-3jw2 krb5kdc[13112](info): set up 2 sockets
Sep 26 12:28:48 dist-test-slave-3jw2 krb5kdc[13112](info): commencing operation
krb5kdc: starting...
W20260926 12:28:49.012694 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:28:49.129284 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:34949
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:34949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:49.239492 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:49.239738 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:49.239778 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:49.243813 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:49.243865 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:49.243883 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:49.243899 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:49.243948 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:49.249130 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:34949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:34949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:49.250111 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:49.251113 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:49.259346 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.259318 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.259429 13125 server_base.cc:1034] running on GCE node
W20260926 12:28:49.259330 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.260142 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:49.261217 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:49.262372 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790425729262354 us; error 27 us; skew 500 ppm
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:49.272403 13125 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:28:49.273289 13125 webserver.cc:460] Webserver started at http://127.7.99.62:42147/ using document root <none> and password file <none>
I20260926 12:28:49.273887 13125 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:49.273991 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:49.274231 13125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:49.276448 13125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "b6ef6ccbc3c34006ac4125008175e945"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "d106707c2166fc7f09e38f26d5b17e4b"
server_key_iv: "3acfc5bd83ffe2ae3a3a697cc473cf3c"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.277032 13125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "b6ef6ccbc3c34006ac4125008175e945"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "d106707c2166fc7f09e38f26d5b17e4b"
server_key_iv: "3acfc5bd83ffe2ae3a3a697cc473cf3c"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.280761 13125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:49.283247 13140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.284512 13125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:49.284657 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "b6ef6ccbc3c34006ac4125008175e945"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "d106707c2166fc7f09e38f26d5b17e4b"
server_key_iv: "3acfc5bd83ffe2ae3a3a697cc473cf3c"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.284763 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:49.301339 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:49.304556 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:49.304751 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:49.311630 13125 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:34949
I20260926 12:28:49.311654 13192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:34949 every 8 connection(s)
I20260926 12:28:49.312626 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:49.315222 13193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:49.315558 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13125
I20260926 12:28:49.315683 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:49.315979 7564 external_mini_cluster.cc:1428] Setting key fb2c5a560b4cd65523c9a50cff9b5461
I20260926 12:28:49.321532 13193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945: Bootstrap starting.
I20260926 12:28:49.323619 13193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:49.324380 13193 log.cc:824] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:49.326761 13193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945: No bootstrap required, opened a new log
I20260926 12:28:49.330432 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:49.317416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.7.99.62:34949)
0926 12:28:49.317813 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:49.317829 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:49.317863 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:28:49.318522 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:49.318544 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:49.318861 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:49.319193 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:49.319969 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.321019 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.321909 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.322380 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.322561 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:49.325244 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:49.325250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:49.325269 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:49.325303 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:49.327234 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.327648 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.327654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.327660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.327797 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.328104 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.328107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.328109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.328637 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:49.328837 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:49.329124 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 12:28:49.329479 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260926 12:28:49.331387 13193 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } }
I20260926 12:28:49.331655 13193 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:49.331723 13193 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ef6ccbc3c34006ac4125008175e945, State: Initialized, Role: FOLLOWER
I20260926 12:28:49.332233 13193 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } }
I20260926 12:28:49.332376 13193 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:49.332456 13193 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:49.332557 13193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:49.333501 13193 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } }
I20260926 12:28:49.333771 13193 leader_election.cc:302] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ef6ccbc3c34006ac4125008175e945; no voters:
I20260926 12:28:49.334020 13193 leader_election.cc:288] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:49.334165 13198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:49.334394 13198 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [term 1 LEADER]: Becoming Leader. State: Replica: b6ef6ccbc3c34006ac4125008175e945, State: Running, Role: LEADER
I20260926 12:28:49.334710 13198 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } }
I20260926 12:28:49.335206 13193 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:49.336524 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ef6ccbc3c34006ac4125008175e945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } } }
I20260926 12:28:49.336671 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:49.336524 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ef6ccbc3c34006ac4125008175e945. Latest consensus state: current_term: 1 leader_uuid: "b6ef6ccbc3c34006ac4125008175e945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef6ccbc3c34006ac4125008175e945" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34949 } } }
I20260926 12:28:49.336805 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:49.337107 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:49.339839 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:49.344839 13209 catalog_manager.cc:1347] Generated new cluster ID: 827d76fc081146e1b7c257f9ef6a5f6e
I20260926 12:28:49.344901 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:49.362212 13209 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:49.363598 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:49.380918 13209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ef6ccbc3c34006ac4125008175e945: Generated new TSK 0
I20260926 12:28:49.381767 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:28:49.406929 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:49.516255 13219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:49.516485 13219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:49.516526 13219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:49.520323 13219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:49.520363 13219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:49.520414 13219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:49.525524 13219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:49.526437 13219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:49.527359 13219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:49.536705 13224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.536715 13227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.536795 13225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.536974 13219 server_base.cc:1034] running on GCE node
I20260926 12:28:49.537354 13219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:49.537930 13219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:49.539086 13219 hybrid_clock.cc:648] HybridClock initialized: now 1790425729539075 us; error 23 us; skew 500 ppm
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:49.541844 13219 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:28:49.542608 13219 webserver.cc:460] Webserver started at http://127.7.99.1:46443/ using document root <none> and password file <none>
I20260926 12:28:49.543107 13219 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:49.543160 13219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:49.543306 13219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:49.545272 13219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "756a4e70c7d74b2aa9153351cd9453ba"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "7c1119f98b4c3ec270f5c1ff774391d1"
server_key_iv: "3e2f36a874b3767a8e08cc8e584670ad"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.545766 13219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "756a4e70c7d74b2aa9153351cd9453ba"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "7c1119f98b4c3ec270f5c1ff774391d1"
server_key_iv: "3e2f36a874b3767a8e08cc8e584670ad"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.549631 13219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:28:49.551880 13234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.553097 13219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:49.553184 13219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "756a4e70c7d74b2aa9153351cd9453ba"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "7c1119f98b4c3ec270f5c1ff774391d1"
server_key_iv: "3e2f36a874b3767a8e08cc8e584670ad"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.553262 13219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:49.568858 13219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:49.571992 13219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:49.572136 13219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:49.572731 13219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:49.573514 13219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:49.573559 13219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.573626 13219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:49.573657 13219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.582477 13219 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:37205
I20260926 12:28:49.582490 13347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:37205 every 8 connection(s)
I20260926 12:28:49.583333 13219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:49.592785 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13219
I20260926 12:28:49.592908 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:49.593209 7564 external_mini_cluster.cc:1428] Setting key 563b33d3a16614e85adfebd55d69bbfb
I20260926 12:28:49.597395 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:49.585171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51061 (local address 127.7.99.62:34949)
0926 12:28:49.585316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:49.585320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:49.586058 (+ 738us) server_negotiation.cc:407] Connection header received
0926 12:28:49.587450 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:49.587453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:49.587508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:49.587574 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:49.588785 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.589325 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.590449 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.590714 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.590830 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:49.593504 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:49.593507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:49.593510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:49.593539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:49.595066 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.595687 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.595689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.595691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.595764 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.596196 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.596201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.596204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.596466 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:49.596574 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:49.597049 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 12:28:49.597190 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:49.598125 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:49.585458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:34949 (local address 127.7.99.1:51061)
0926 12:28:49.585885 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:49.585925 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:49.587238 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:49.587745 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:49.587751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:49.588084 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:49.588639 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:49.588653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.589451 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.589456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:49.590321 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:49.590330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.590867 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.590871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:49.591037 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:49.591634 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:49.591650 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:49.593319 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:49.595186 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:49.595195 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:49.595216 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:49.595583 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:49.595855 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:49.595861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:49.595865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:49.596062 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:49.596576 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:49.596582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:49.596808 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 12:28:49.596973 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 12:28:49.597224 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260926 12:28:49.599298 13348 heartbeater.cc:347] Connected to a master server at 127.7.99.62:34949
I20260926 12:28:49.599560 13348 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:49.600086 13348 heartbeater.cc:511] Master 127.7.99.62:34949 requested a full tablet report, sending...
I20260926 12:28:49.601601 13157 ts_manager.cc:194] Registered new tserver with Master: 756a4e70c7d74b2aa9153351cd9453ba (127.7.99.1:37205)
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:28:49.602914 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:51061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:28:49.619148 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:49.727035 13353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:49.727279 13353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:49.727320 13353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:49.731338 13353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:49.731396 13353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:49.731480 13353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:49.736886 13353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:49.737946 13353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:49.738921 13353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:28:49.748277 13353 server_base.cc:1034] running on GCE node
W20260926 12:28:49.748342 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.748298 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.748288 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.748899 13353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:49.749536 13353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:49.750718 13353 hybrid_clock.cc:648] HybridClock initialized: now 1790425729750707 us; error 31 us; skew 500 ppm
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:49.753412 13353 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:28:49.754271 13353 webserver.cc:460] Webserver started at http://127.7.99.2:41405/ using document root <none> and password file <none>
I20260926 12:28:49.754757 13353 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:49.754810 13353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:49.754962 13353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:49.756979 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "14734a1d75ed4c3f88c8d3fa203dc832"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "1bb5c3edcabea6f6ec88fe7a875cfea8"
server_key_iv: "84204546e48580ad0fa6ab09e44a6ce4"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.757541 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "14734a1d75ed4c3f88c8d3fa203dc832"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "1bb5c3edcabea6f6ec88fe7a875cfea8"
server_key_iv: "84204546e48580ad0fa6ab09e44a6ce4"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.761065 13353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:49.763314 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.764519 13353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:49.764638 13353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "14734a1d75ed4c3f88c8d3fa203dc832"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "1bb5c3edcabea6f6ec88fe7a875cfea8"
server_key_iv: "84204546e48580ad0fa6ab09e44a6ce4"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.764734 13353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:49.775576 13353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:49.778676 13353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:49.778815 13353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:49.779398 13353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:49.780223 13353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:49.780282 13353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.780345 13353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:49.780381 13353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.788695 13353 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:33215
I20260926 12:28:49.788721 13481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:33215 every 8 connection(s)
I20260926 12:28:49.789546 13353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:49.794729 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13353
I20260926 12:28:49.794858 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:49.795151 7564 external_mini_cluster.cc:1428] Setting key 319fe9c7e0948cdcc6a2d450ad76d482
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:49.803481 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:49.791215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:54275 (local address 127.7.99.62:34949)
0926 12:28:49.791345 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:49.791349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:49.792148 (+ 799us) server_negotiation.cc:407] Connection header received
0926 12:28:49.793491 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:49.793495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:49.793549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:49.793657 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:49.794728 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.795390 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.796730 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.796959 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.797078 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:49.799719 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:49.799726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:49.799732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:49.799764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:49.801460 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.801924 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.801931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.801933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.802035 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:49.802348 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:49.802353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:49.802356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:49.802605 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:49.802720 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:49.803179 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 12:28:49.803307 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:28:49.804255 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:49.791580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:34949 (local address 127.7.99.2:54275)
0926 12:28:49.792010 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:49.792050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:49.793285 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:49.793814 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:49.793820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:49.794143 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:49.794583 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:49.794596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.795513 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.795517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:49.796608 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:49.796616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:49.797106 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:49.797110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:49.797271 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:49.797837 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:49.797852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:49.799551 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:49.801568 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:49.801573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:49.801583 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:49.801821 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:49.802132 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:49.802134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:49.802135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:49.802253 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:49.802710 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:49.802714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:49.802923 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 12:28:49.803083 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:28:49.803320 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:28:49.805403 13482 heartbeater.cc:347] Connected to a master server at 127.7.99.62:34949
I20260926 12:28:49.805651 13482 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:49.806095 13482 heartbeater.cc:511] Master 127.7.99.62:34949 requested a full tablet report, sending...
I20260926 12:28:49.807053 13157 ts_manager.cc:194] Registered new tserver with Master: 14734a1d75ed4c3f88c8d3fa203dc832 (127.7.99.2:33215)
I20260926 12:28:49.807672 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:54275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:28:49.823244 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:49.933225 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:49.933525 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:49.933588 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:49.937855 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:49.937906 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:49.937961 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:49.942976 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:49.943996 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:49.945147 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:49.954523 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:49.954537 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.954622 13487 server_base.cc:1034] running on GCE node
W20260926 12:28:49.954519 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:49.955139 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:49.955809 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:49.956959 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790425729956909 us; error 62 us; skew 500 ppm
Sep 26 12:28:49 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:49.959782 13487 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:28:49.960659 13487 webserver.cc:460] Webserver started at http://127.7.99.3:37911/ using document root <none> and password file <none>
I20260926 12:28:49.961205 13487 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:49.961292 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:49.961532 13487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:49.963404 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "ac50277db7d144b5b0d950f5df73d883"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "94fbec9c332e321f72856f4364bb3839"
server_key_iv: "a879c19c639d335a63ed5cda85e7201f"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.963944 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "ac50277db7d144b5b0d950f5df73d883"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "94fbec9c332e321f72856f4364bb3839"
server_key_iv: "a879c19c639d335a63ed5cda85e7201f"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.967360 13487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:28:49.969842 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.971010 13487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 12:28:49.971143 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "ac50277db7d144b5b0d950f5df73d883"
format_stamp: "Formatted at 2026-09-26 12:28:49 on dist-test-slave-3jw2"
server_key: "94fbec9c332e321f72856f4364bb3839"
server_key_iv: "a879c19c639d335a63ed5cda85e7201f"
server_key_version: "encryptionkey@0"
I20260926 12:28:49.971251 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:49.983865 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:49.987063 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:49.987260 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:49.987847 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:49.988760 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:49.988829 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.988902 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:49.988955 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:49.997706 13487 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:43753
I20260926 12:28:49.997726 13615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:43753 every 8 connection(s)
I20260926 12:28:49.998616 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:49.999389 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13487
I20260926 12:28:49.999511 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:49.999804 7564 external_mini_cluster.cc:1428] Setting key bed1c6b61904183558af45694e911213
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425729, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:28:50.011531 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:50.000658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:51689 (local address 127.7.99.62:34949)
0926 12:28:50.000827 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.000831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.001546 (+ 715us) server_negotiation.cc:407] Connection header received
0926 12:28:50.002935 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.002938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.002992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.003070 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.004145 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.004705 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.005764 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.005981 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.006109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:50.008386 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:50.008394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:50.008396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:50.008417 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:50.009631 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.010091 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.010094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.010096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.010170 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.010488 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.010491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.010493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.010690 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:50.010792 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:50.011250 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 12:28:50.011358 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:28:50.012300 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:50.000939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:34949 (local address 127.7.99.3:51689)
0926 12:28:50.001417 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:50.001458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:50.002739 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:50.003224 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:50.003231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:50.003563 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:50.004012 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:50.004025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.004823 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.004828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:50.005650 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:50.005659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.006139 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.006143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:50.006299 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:50.006794 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:50.006808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:50.008236 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:50.009740 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:50.009746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:50.009758 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:50.009992 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:50.010263 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:50.010265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:50.010267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:50.010392 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:50.010799 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:50.010808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:50.011035 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 12:28:50.011210 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 12:28:50.011490 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260926 12:28:50.013288 13616 heartbeater.cc:347] Connected to a master server at 127.7.99.62:34949
I20260926 12:28:50.013504 13616 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:50.013954 13616 heartbeater.cc:511] Master 127.7.99.62:34949 requested a full tablet report, sending...
I20260926 12:28:50.014815 13157 ts_manager.cc:194] Registered new tserver with Master: ac50277db7d144b5b0d950f5df73d883 (127.7.99.3:43753)
I20260926 12:28:50.015298 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:51689
I20260926 12:28:50.025334 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/test-admin.keytab.
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.040014 7564 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 12:28:50.042691 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:50.041257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.7.99.1:37205)
0926 12:28:50.041599 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.041609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.041630 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:28:50.041674 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.041678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.041841 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.042014 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.042480 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
W20260926 12:28:50.042762 7564 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.043711 7564 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.044637 7564 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:50.053669 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:50.045422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.7.99.1:37205)
0926 12:28:50.045513 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.045516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.045584 (+ 68us) server_negotiation.cc:407] Connection header received
0926 12:28:50.045664 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.045666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.045700 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.045763 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.046230 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.046955 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.047710 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.048051 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.048125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:50.050180 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:50.050187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:50.050198 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:50.050218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:50.051861 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.052366 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.052371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.052373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.052444 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.052749 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.052754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.052756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.052976 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:50.053196 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:50.053357 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 12:28:50.053526 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 12:28:50.057327 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:50.056245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.7.99.1:37205)
0926 12:28:50.056349 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.056353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.056522 (+ 169us) server_negotiation.cc:407] Connection header received
0926 12:28:50.056628 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.056631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.056671 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.056741 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.057222 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47582: BlockingRecv error: recv got EOF from 127.0.0.1:47582 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 12:28:50.057440 7564 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.058338 7564 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.059216 7564 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:50.066790 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:50.059914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.7.99.1:37205)
0926 12:28:50.060036 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.060039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.060079 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:28:50.060190 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.060193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.060233 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.060293 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.060735 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.061228 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.061953 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.062160 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.062229 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:50.064299 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:50.064302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:50.064304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:50.064321 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:50.065475 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.065817 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.065819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.065820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.065883 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.066122 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.066124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.066125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.066257 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:50.066345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:50.066517 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 12:28:50.066621 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 12:28:50.069216 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:50.068304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.7.99.1:37205)
0926 12:28:50.068412 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.068416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.068542 (+ 126us) server_negotiation.cc:407] Connection header received
0926 12:28:50.068673 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.068675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.068709 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.068788 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.069122 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47590: BlockingRecv error: recv got EOF from 127.0.0.1:47590 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:28:50.069312 7564 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425700812969-7564-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.070219 7564 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:50.071046 7564 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:28:50.078490 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:50.071701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.7.99.1:37205)
0926 12:28:50.071818 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:50.071821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:50.071847 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:28:50.071932 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:50.071934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:50.071971 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:50.072028 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:50.072520 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.072975 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.073665 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:50.073854 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:50.073920 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:50.076020 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:50.076023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:50.076025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:50.076062 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:50.077223 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.077561 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.077563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.077564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.077625 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:50.077833 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:50.077835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:50.077836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:50.077984 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:50.078066 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:50.078238 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 12:28:50.078340 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:50.079345 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13219
I20260926 12:28:50.085013 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13353
I20260926 12:28:50.090487 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13487
I20260926 12:28:50.095566 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13125
2026-09-26T12:28:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2117 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13636](info): set up 2 sockets
Sep 26 12:28:50 dist-test-slave-3jw2 krb5kdc[13636](info): commencing operation
krb5kdc: starting...
W20260926 12:28:51.128826 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab.
Entry for principal oryx/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab.
I20260926 12:28:51.249007 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43561
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab
--principal=oryx/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:51.356207 13649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:51.356457 13649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:51.356496 13649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:51.360462 13649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:51.360517 13649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:51.360536 13649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:51.360584 13649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:51.360615 13649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:51.366022 13649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab
--principal=oryx/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:51.367082 13649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:51.368132 13649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:51.376827 13654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:51.376752 13655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:51.376749 13657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:51.376809 13649 server_base.cc:1034] running on GCE node
I20260926 12:28:51.377592 13649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:51.378629 13649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:51.379791 13649 hybrid_clock.cc:648] HybridClock initialized: now 1790425731379763 us; error 40 us; skew 500 ppm
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:51.382391 13649 init.cc:377] Logged in from keytab as oryx/127.7.99.62@KRBTEST.COM (short username oryx)
I20260926 12:28:51.383230 13649 webserver.cc:460] Webserver started at http://127.7.99.62:38209/ using document root <none> and password file <none>
I20260926 12:28:51.383754 13649 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:51.383841 13649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:51.384076 13649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:51.386055 13649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "08ec1828ecc2448ca760196b3cd9599e"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "5cf99de6b57c55b90bfd554a054e4f8f"
server_key_iv: "1b2ac9da34200e1018670da1fea36e61"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.386601 13649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "08ec1828ecc2448ca760196b3cd9599e"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "5cf99de6b57c55b90bfd554a054e4f8f"
server_key_iv: "1b2ac9da34200e1018670da1fea36e61"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.390208 13649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:28:51.392663 13664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.394027 13649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:51.394161 13649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "08ec1828ecc2448ca760196b3cd9599e"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "5cf99de6b57c55b90bfd554a054e4f8f"
server_key_iv: "1b2ac9da34200e1018670da1fea36e61"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.394268 13649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:51.424023 13649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:51.427080 13649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:51.427279 13649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:51.434648 13649 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43561
I20260926 12:28:51.435025 13716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43561 every 8 connection(s)
I20260926 12:28:51.435637 13649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:51.438602 13717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:51.444470 13717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Bootstrap starting.
I20260926 12:28:51.445129 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13649
I20260926 12:28:51.445271 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:51.445595 7564 external_mini_cluster.cc:1428] Setting key 76d3b7cc9f567f9321d77f602f6465a5
I20260926 12:28:51.447563 13717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:51.448408 13717 log.cc:824] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:51.450515 13717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: No bootstrap required, opened a new log
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:51.455500 13717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } }
I20260926 12:28:51.455770 13717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:51.455848 13717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ec1828ecc2448ca760196b3cd9599e, State: Initialized, Role: FOLLOWER
I20260926 12:28:51.456435 13717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } }
I20260926 12:28:51.456624 13717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:51.456707 13717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:51.456838 13717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:51.457755 13717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } }
I20260926 12:28:51.458108 13717 leader_election.cc:302] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ec1828ecc2448ca760196b3cd9599e; no voters:
I20260926 12:28:51.458444 13717 leader_election.cc:288] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:51.458508 13722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:51.458740 13722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 LEADER]: Becoming Leader. State: Replica: 08ec1828ecc2448ca760196b3cd9599e, State: Running, Role: LEADER
I20260926 12:28:51.459091 13722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } }
I20260926 12:28:51.459807 13717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:51.460430 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec1828ecc2448ca760196b3cd9599e. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } }
I20260926 12:28:51.460599 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:51.460927 13723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } }
I20260926 12:28:51.461059 13723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:51.461092 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:51.447018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.7.99.62:43561)
0926 12:28:51.447465 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:51.447480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:51.447514 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:28:51.448155 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:51.448225 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:51.448576 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:51.448912 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:51.449810 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.450834 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.452095 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.452597 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.452709 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:51.455774 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:51.455782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:51.455803 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:51.455839 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:51.457627 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.458137 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.458143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.458148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.458294 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.458604 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.458607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.458609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.459114 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:51.459320 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:51.459612 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 12:28:51.460052 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260926 12:28:51.461905 13728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:51.464900 13728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:51.470320 13728 catalog_manager.cc:1347] Generated new cluster ID: 87bcdefce27148d6aacf059caa76cbbf
I20260926 12:28:51.470387 13728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:51.509507 13728 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:51.510469 13728 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:51.518227 13728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Generated new TSK 0
I20260926 12:28:51.518901 13728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
I20260926 12:28:51.551328 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:51.663381 13743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:51.663626 13743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:51.663667 13743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:51.667626 13743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:51.667698 13743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:51.667781 13743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:51.672916 13743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:51.673977 13743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:51.674979 13743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:51.684216 13749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:51.684152 13748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:51.684187 13751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:51.684597 13743 server_base.cc:1034] running on GCE node
I20260926 12:28:51.684998 13743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:51.685640 13743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:51.686862 13743 hybrid_clock.cc:648] HybridClock initialized: now 1790425731686832 us; error 47 us; skew 500 ppm
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:51.689546 13743 init.cc:377] Logged in from keytab as oryx/127.7.99.1@KRBTEST.COM (short username oryx)
I20260926 12:28:51.690392 13743 webserver.cc:460] Webserver started at http://127.7.99.1:33803/ using document root <none> and password file <none>
I20260926 12:28:51.690956 13743 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:51.691046 13743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:51.691272 13743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:51.693390 13743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "79ed588c4ac14303bd5c9c25d8b5afe5"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "e1574881cedc8ee3b810f671150134bb"
server_key_iv: "155fea2865522285e5ea4b6329d64798"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.693955 13743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "79ed588c4ac14303bd5c9c25d8b5afe5"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "e1574881cedc8ee3b810f671150134bb"
server_key_iv: "155fea2865522285e5ea4b6329d64798"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.697732 13743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:28:51.700060 13758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.701445 13743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:51.701581 13743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "79ed588c4ac14303bd5c9c25d8b5afe5"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "e1574881cedc8ee3b810f671150134bb"
server_key_iv: "155fea2865522285e5ea4b6329d64798"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.701704 13743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:51.734771 13743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:51.737742 13743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:51.737955 13743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:51.738554 13743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:51.739385 13743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:51.739455 13743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.739527 13743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:51.739570 13743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.748287 13743 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:38113
I20260926 12:28:51.748306 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:38113 every 8 connection(s)
I20260926 12:28:51.749163 13743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:51.757655 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13743
I20260926 12:28:51.757784 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:51.758140 7564 external_mini_cluster.cc:1428] Setting key cb7d62abe4f6a4c9923adc5b3f2b1e91
I20260926 12:28:51.762420 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:51.750925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:35875 (local address 127.7.99.62:43561)
0926 12:28:51.751063 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:51.751067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:51.752059 (+ 992us) server_negotiation.cc:407] Connection header received
0926 12:28:51.752905 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:51.752907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:51.752953 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:51.753026 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:51.754129 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.754688 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.755766 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.756028 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.756120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:51.758619 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:51.758641 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:51.758648 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:51.758675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:51.760284 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.760871 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.760877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.760882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.760972 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.761325 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.761331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.761336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.761576 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:51.761707 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:51.762203 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 12:28:51.762301 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:28:51.763574 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:51.751293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:35875)
0926 12:28:51.751910 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:51.751949 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:51.752730 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:51.753205 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:51.753211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:51.753537 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:51.753991 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:51.754004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.754790 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.754793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:51.755653 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:51.755661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.756215 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.756225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:51.756363 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:51.756865 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:51.756880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:51.758433 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:51.760459 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:51.760465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:51.760483 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:51.760780 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:51.761092 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:51.761095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:51.761097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:51.761228 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:51.761681 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:51.761689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:51.761916 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 12:28:51.762592 (+ 676us) client_negotiation.cc:240] Negotiation successful
0926 12:28:51.762839 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":110,"threads_started":1}
I20260926 12:28:51.764722 13872 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:51.765027 13872 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:51.765641 13872 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:51.767217 13681 ts_manager.cc:194] Registered new tserver with Master: 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
WARNING: no policy specified for oryx/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:28:51.768430 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:35875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
I20260926 12:28:51.787204 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:51.894590 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:51.894843 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:51.894917 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:51.899065 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:51.899113 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:51.899209 13877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:51.904320 13877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:51.905284 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:51.906363 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:51.915635 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:51.915776 13877 server_base.cc:1034] running on GCE node
W20260926 12:28:51.915711 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:51.915673 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:51.916376 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:51.917024 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:51.918217 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790425731918186 us; error 43 us; skew 500 ppm
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:51.920814 13877 init.cc:377] Logged in from keytab as oryx/127.7.99.2@KRBTEST.COM (short username oryx)
I20260926 12:28:51.921629 13877 webserver.cc:460] Webserver started at http://127.7.99.2:38625/ using document root <none> and password file <none>
I20260926 12:28:51.922216 13877 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:51.922302 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:51.922521 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:51.924599 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "32fdeb060b434521b8aa5571c0e82196"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "ec16891ea3d482d286879eb9b4227168"
server_key_iv: "3f64a09a5de5cd5167e6cb695ce75dc2"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.925156 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "32fdeb060b434521b8aa5571c0e82196"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "ec16891ea3d482d286879eb9b4227168"
server_key_iv: "3f64a09a5de5cd5167e6cb695ce75dc2"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.928726 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:51.931077 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.932314 13877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:51.932437 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "32fdeb060b434521b8aa5571c0e82196"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "ec16891ea3d482d286879eb9b4227168"
server_key_iv: "3f64a09a5de5cd5167e6cb695ce75dc2"
server_key_version: "encryptionkey@0"
I20260926 12:28:51.932554 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:51.948545 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:51.951323 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:51.951509 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:51.952066 13877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:51.952924 13877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:51.952993 13877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.953069 13877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:51.953121 13877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:51.961879 13877 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:44669
I20260926 12:28:51.961897 14005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:44669 every 8 connection(s)
I20260926 12:28:51.962849 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
Sep 26 12:28:51 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:51.972800 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 13877
I20260926 12:28:51.972922 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:51.973197 7564 external_mini_cluster.cc:1428] Setting key c63ca33489fea8f8acadb4939e085b42
I20260926 12:28:51.976215 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:51.964599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:50825 (local address 127.7.99.62:43561)
0926 12:28:51.964739 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:51.964743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:51.965743 (+ 1000us) server_negotiation.cc:407] Connection header received
0926 12:28:51.966622 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:51.966625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:51.966682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:51.966781 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:51.967892 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.968487 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.969596 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.969816 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.969923 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:51.972340 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:51.972344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:51.972346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:51.972368 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:51.974072 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.974625 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.974628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.974630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.974712 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:51.975089 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:51.975091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:51.975092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:51.975278 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:51.975347 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:51.975965 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 12:28:51.976056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:51.977393 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:51.964962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:50825)
0926 12:28:51.965604 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:51.965645 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:51.966420 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:51.966941 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:51.966947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:51.967292 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:51.967756 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:51.967768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.968606 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.968614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:51.969481 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:51.969489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:51.969965 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:51.969969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:51.970128 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:51.970730 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:51.970746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:51.972133 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:51.974175 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:51.974179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:51.974190 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:51.974510 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:51.974811 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:51.974815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:51.974817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:51.974988 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:51.975379 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:51.975389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:51.975722 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 12:28:51.975931 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 12:28:51.976308 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":100,"threads_started":1}
I20260926 12:28:51.978539 14006 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:51.978766 14006 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:51.979224 14006 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:51.980310 13681 ts_manager.cc:194] Registered new tserver with Master: 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:51.980872 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:50825
WARNING: no policy specified for oryx/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
I20260926 12:28:51.999091 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:52.104061 14011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:52.104341 14011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:52.104411 14011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:52.108526 14011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:52.108604 14011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:52.108690 14011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:52.113682 14011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:52.114605 14011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:52.115536 14011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.124408 14019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.124395 14017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.124470 14016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.124701 14011 server_base.cc:1034] running on GCE node
I20260926 12:28:52.125141 14011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.125757 14011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.126935 14011 hybrid_clock.cc:648] HybridClock initialized: now 1790425732126916 us; error 28 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.129572 14011 init.cc:377] Logged in from keytab as oryx/127.7.99.3@KRBTEST.COM (short username oryx)
I20260926 12:28:52.130281 14011 webserver.cc:460] Webserver started at http://127.7.99.3:45823/ using document root <none> and password file <none>
I20260926 12:28:52.130805 14011 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.130864 14011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.131014 14011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:52.132915 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "b8e1ac7a94414f268bc12640cdccba09"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "7d5ac97a3cc128bcb36deca4cec95bc8"
server_key_iv: "d74d1b4ac0ae46571bc10a71e817cbb0"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.133394 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "b8e1ac7a94414f268bc12640cdccba09"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "7d5ac97a3cc128bcb36deca4cec95bc8"
server_key_iv: "d74d1b4ac0ae46571bc10a71e817cbb0"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.137059 14011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:52.139508 14026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.141063 14011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:52.141202 14011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "b8e1ac7a94414f268bc12640cdccba09"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "7d5ac97a3cc128bcb36deca4cec95bc8"
server_key_iv: "d74d1b4ac0ae46571bc10a71e817cbb0"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.141333 14011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.164279 14011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:52.166970 14011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:52.167160 14011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:52.167745 14011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:52.168617 14011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:52.168663 14011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.168728 14011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:52.168771 14011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.177322 14011 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:34519
I20260926 12:28:52.177342 14139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:34519 every 8 connection(s)
I20260926 12:28:52.178198 14011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:52.184448 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14011
I20260926 12:28:52.184576 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:52.184881 7564 external_mini_cluster.cc:1428] Setting key 5770e35016eb02969947c68ee4e371e2
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:52.191236 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.179822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:54995 (local address 127.7.99.62:43561)
0926 12:28:52.179947 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.179951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.181091 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 12:28:52.181951 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.181955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.182012 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.182120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.183257 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.183776 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.185301 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.185587 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.185683 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.188130 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.188132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.188134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.188154 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.189381 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.189891 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.189893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.189894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.189963 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.190275 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.190277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.190278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.190473 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.190579 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.191009 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.191116 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:52.192063 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.180294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:54995)
0926 12:28:52.180935 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.180983 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.181752 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.182286 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.182293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.182637 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.183119 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.183131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.183892 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.183896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.185179 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.185191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.185763 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.185767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.185938 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.186509 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.186525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.187993 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.189503 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.189507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.189517 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.189797 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.190039 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.190041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.190042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.190158 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.190579 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.190583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.190803 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.190947 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.191173 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":114,"threads_started":1}
I20260926 12:28:52.193289 14140 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:52.193504 14140 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:52.193940 14140 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:52.194904 13681 ts_manager.cc:194] Registered new tserver with Master: b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519)
I20260926 12:28:52.195416 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:54995
I20260926 12:28:52.198496 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.61.keytab.
Entry for principal oryx/127.7.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.61.keytab.
I20260926 12:28:52.223171 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:44375
--webserver_interface=127.7.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.61.keytab
--principal=oryx/127.7.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43561,127.7.99.61:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:52.328804 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:52.329062 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:52.329157 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:52.333128 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:52.333195 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:52.333230 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:52.333282 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:52.333316 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:52.338348 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43561,127.7.99.61:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.61.keytab
--principal=oryx/127.7.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
W20260926 12:28:52.338558 14145 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.99.62:43561,127.7.99.61:44375'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 12:28:52.339314 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:52.340338 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.348431 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.348439 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.348479 14145 server_base.cc:1034] running on GCE node
W20260926 12:28:52.348588 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.349179 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.350167 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.351320 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790425732351299 us; error 40 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.353945 14145 init.cc:377] Logged in from keytab as oryx/127.7.99.61@KRBTEST.COM (short username oryx)
I20260926 12:28:52.354766 14145 webserver.cc:460] Webserver started at http://127.7.99.61:33559/ using document root <none> and password file <none>
I20260926 12:28:52.355275 14145 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.355331 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.355531 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:52.357472 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/instance:
uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "da48ca6921b84758962631c86bf3ceaa"
server_key_iv: "07fb6ecd2084e5f082a9844a89bed11c"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.358003 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal/instance:
uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "da48ca6921b84758962631c86bf3ceaa"
server_key_iv: "07fb6ecd2084e5f082a9844a89bed11c"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.361694 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:52.364114 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.365379 14145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:52.365506 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "da48ca6921b84758962631c86bf3ceaa"
server_key_iv: "07fb6ecd2084e5f082a9844a89bed11c"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.365620 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.388110 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:52.391281 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:52.391479 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:52.398375 14145 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.61:44375
I20260926 12:28:52.398674 14212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.61:44375 every 8 connection(s)
I20260926 12:28:52.399333 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
I20260926 12:28:52.402546 14213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:52.406484 14213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:28:52.408663 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14145
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.61@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
I20260926 12:28:52.421190 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.408319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.7.99.62:43561)
0926 12:28:52.408442 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.408446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.409356 (+ 910us) server_negotiation.cc:407] Connection header received
0926 12:28:52.410393 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.410396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.410440 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.410542 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.411632 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.412464 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.413920 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.414253 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.414354 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.417157 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.417160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.417162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.417183 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.418710 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.419252 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.419255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.419257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.419369 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.419705 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.419708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.419709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.420008 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.420123 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.420807 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.420997 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:28:52.422021 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.409617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.7.99.61:44375)
0926 12:28:52.409881 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.409892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.409931 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:28:52.410214 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.410231 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.410443 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.410628 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.411476 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.412727 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.413859 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.414412 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.414531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.417580 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.417589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.417602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.417628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.419243 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.419655 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.419662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.419664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.419743 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.419996 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.419998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.419999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.420576 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.420773 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.421017 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.421141 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260926 12:28:52.422021 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.408714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56662)
0926 12:28:52.409212 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.409261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.410203 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.410637 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.410642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.411157 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.411507 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.411519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.412592 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.412594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.413744 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.413753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.414390 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.414393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.414536 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.415203 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.415224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.416984 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.418822 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.418831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.418840 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.419156 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.419469 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.419472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.419473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.419592 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.420153 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.420159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.420576 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.420749 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.421086 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":46,"spinlock_wait_cycles":15744,"thread_start_us":149,"threads_started":1}
I20260926 12:28:52.423678 14213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.61@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
I20260926 12:28:52.430892 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.424124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:59978)
0926 12:28:52.424237 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.424250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.424318 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.424509 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.424512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.424653 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.424720 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.424725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.425397 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.425403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.425913 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.425919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.426338 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.426341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.426492 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.426885 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.426896 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.427969 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.429571 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.429575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.429576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.429748 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.429994 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.430000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.430002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.430075 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.430414 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.430417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.430503 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.430622 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.430737 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":33}
I20260926 12:28:52.430927 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.424199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.7.99.61:44375)
0926 12:28:52.424294 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.424297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.424308 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:28:52.424387 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.424391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.424427 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.424501 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.424818 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.425282 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.426020 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.426241 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.426340 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.428089 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.428093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.428095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.428119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.429458 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.429842 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.429845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.429846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.429910 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.430179 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.430182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.430183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.430323 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.430414 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.430699 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.430794 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":35}
I20260926 12:28:52.433844 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Bootstrap starting.
I20260926 12:28:52.436153 14213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:52.437057 14213 log.cc:824] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:52.438834 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: No bootstrap required, opened a new log
I20260926 12:28:52.441911 14213 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } }
I20260926 12:28:52.442096 14213 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:52.442150 14213 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18ca25f6ac448fa8e21839e6a4ba216, State: Initialized, Role: FOLLOWER
I20260926 12:28:52.442538 14213 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } }
I20260926 12:28:52.442977 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } } }
I20260926 12:28:52.443095 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:52.443616 14213 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:28:52.448441 14230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:28:52.448637 14230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:28:52.449404 14145 master_runner.cc:428] Detected that this master d18ca25f6ac448fa8e21839e6a4ba216 is joining an existing cluster
I20260926 12:28:52.449474 14145 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.99.61:44375
I20260926 12:28:52.459317 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.451989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.7.99.61:44375)
0926 12:28:52.452072 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.452076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.452087 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:28:52.452117 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.452120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.452179 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.452253 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.452767 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.453253 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.454313 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.454650 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.454755 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.455784 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.455787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.455790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.455811 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.457623 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458107 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.458113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.458116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.458197 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458483 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.458486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.458488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.458700 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.458785 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.459054 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.459200 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 12:28:52.459741 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.451627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:59980)
0926 12:28:52.451931 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.451944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.452033 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.452281 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.452284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.452519 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.452648 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.452655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.453387 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.453390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.454197 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.454205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.454770 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.454773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.454863 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.455351 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.455362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.455650 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.457732 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.457736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.457738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.457999 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.458298 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.458304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.458405 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.458816 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.458818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.458886 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.459456 (+ 570us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.459580 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260926 12:28:52.460207 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.451552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56664)
0926 12:28:52.451798 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.451813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.451895 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.452374 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.452376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.452585 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.452669 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.452674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.453699 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.453702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.454478 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.454484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.455016 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.455020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.455125 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.455550 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.455564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.455932 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.458429 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.458437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.458674 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.458962 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.458973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.459077 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.459512 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.459515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.459594 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.459722 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.459853 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":32512,"thread_start_us":89,"threads_started":1}
I20260926 12:28:52.460229 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.451877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.7.99.62:43561)
0926 12:28:52.452098 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.452103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.452113 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:28:52.452143 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.452147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.452210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.452277 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.452823 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.453592 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.454580 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.454897 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.455014 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.456380 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.456387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.456392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.456425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.458331 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.458775 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.458779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.458780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.458864 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.459181 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.459184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.459186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.459379 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.459469 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.459806 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.460036 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.62@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
I20260926 12:28:52.473763 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.464600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.7.99.61:44375)
0926 12:28:52.464712 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.464716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.465024 (+ 308us) server_negotiation.cc:407] Connection header received
0926 12:28:52.465195 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.465197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.465236 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.465300 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.465995 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.466440 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.467641 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.467864 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.467958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.470323 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.470330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.470333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.470363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.471858 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.472451 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.472458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.472461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.472543 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.472847 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.472852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.472854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.473021 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.473135 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.473509 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.473615 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:52.473795 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.464573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:59992)
0926 12:28:52.464884 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.464935 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.465065 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.465350 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.465357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.465774 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.465889 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.465896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.466536 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.466539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.467497 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.467507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.467982 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.467986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.468103 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.468578 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.468592 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.470186 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.471997 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.472006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.472012 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.472347 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.472644 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.472649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.472651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.472745 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.473135 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.473143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.473271 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.473431 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.473626 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 12:28:52.475435 13681 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.99.61:44375: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
member_type: NON_VOTER
last_known_addr {
host: "127.7.99.61"
port: 44375
}
attrs {
promote: true
}
}
dest_uuid: "08ec1828ecc2448ca760196b3cd9599e"
cas_config_opid_index: -1
I20260926 12:28:52.476183 13681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } }
I20260926 12:28:52.478789 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:52.480026 14187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 12:28:52.478472 14240 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } }
I20260926 12:28:52.481920 14240 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.99.61:44375
I20260926 12:28:52.482198 14240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:52.482422 14145 master.cc:537] Master@127.7.99.61:44375 shutting down...
WARNING: no policy specified for oryx/127.7.99.60@KRBTEST.COM; defaulting to no policy
W20260926 12:28:52.484974 13668 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 12:28:52.485917 13668 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e -> Peer d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61:44375): Couldn't send request to peer d18ca25f6ac448fa8e21839e6a4ba216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:28:52.486492 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } }
I20260926 12:28:52.486604 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.60@KRBTEST.COM" created.
I20260926 12:28:52.487079 14145 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:52.487334 14145 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:52.487434 14145 tablet_replica.cc:330] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: stopping tablet replica
I20260926 12:28:52.492448 14145 master.cc:552] Master@127.7.99.61:44375 shutdown complete.
I20260926 12:28:52.492499 14145 master_runner.cc:305] Clearing existing system tablet
I20260926 12:28:52.492738 14145 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.492789 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.494030 14145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 12:28:52.494931 14247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.495201 14145 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.495316 14145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.495345 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "da48ca6921b84758962631c86bf3ceaa"
server_key_iv: "07fb6ecd2084e5f082a9844a89bed11c"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.497355 14145 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:52.500209 14145 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:28:52.500294 14145 log.cc:1196] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 12:28:52.500535 14145 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Deleting consensus metadata
I20260926 12:28:52.500696 14145 master_runner.cc:315] Copying system tablet from 127.7.99.62:43561
I20260926 12:28:52.501778 14145 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: tablet copy: Beginning tablet copy session from remote peer at address 127.7.99.62:43561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.60.keytab.
Entry for principal oryx/127.7.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.60.keytab.
I20260926 12:28:52.503971 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:34453
--webserver_interface=127.7.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.60.keytab
--principal=oryx/127.7.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:28:52.511214 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.502374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.7.99.62:43561)
0926 12:28:52.502527 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.502551 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.502844 (+ 293us) server_negotiation.cc:407] Connection header received
0926 12:28:52.502910 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.502913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.502980 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.503105 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.503556 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.504573 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.505797 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.506145 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.506220 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.507615 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.507619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.507622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.507645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.509433 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.509874 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.509877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.509879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.509950 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.510259 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.510262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.510264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.510452 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.510519 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.510909 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.511055 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:52.511902 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.502392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56668)
0926 12:28:52.502688 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.502703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.502829 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.503128 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.503131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.503328 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.503423 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.503429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.504696 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.504700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.505662 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.505672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.506370 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.506373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.506457 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.507051 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.507066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.507473 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.509548 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.509551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.509554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.509775 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.510062 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.510065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.510067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.510149 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.510602 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.510605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.510691 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.511570 (+ 879us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.511706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
I20260926 12:28:52.512823 13701 tablet_copy_service.cc:140] P 08ec1828ecc2448ca760196b3cd9599e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d18ca25f6ac448fa8e21839e6a4ba216 ({username='oryx', principal='oryx/127.7.99.61@KRBTEST.COM'} at 127.0.0.1:56668)
I20260926 12:28:52.513032 13701 tablet_copy_service.cc:161] P 08ec1828ecc2448ca760196b3cd9599e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d18ca25f6ac448fa8e21839e6a4ba216 at {username='oryx', principal='oryx/127.7.99.61@KRBTEST.COM'} at 127.0.0.1:56668: session id = d18ca25f6ac448fa8e21839e6a4ba216-00000000000000000000000000000000
I20260926 12:28:52.515264 13701 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:28:52.516878 14145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:28:52.518980 14145 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: tablet copy: Starting download of 0 data blocks...
I20260926 12:28:52.519172 14145 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: tablet copy: Starting download of 1 WAL segments...
I20260926 12:28:52.520902 14145 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:28:52.522840 13701 tablet_copy_service.cc:342] P 08ec1828ecc2448ca760196b3cd9599e: Request end of tablet copy session d18ca25f6ac448fa8e21839e6a4ba216-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.99.61@KRBTEST.COM'} at 127.0.0.1:56668
I20260926 12:28:52.522944 13701 tablet_copy_service.cc:433] P 08ec1828ecc2448ca760196b3cd9599e: ending tablet copy session d18ca25f6ac448fa8e21839e6a4ba216-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d18ca25f6ac448fa8e21839e6a4ba216
I20260926 12:28:52.529969 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.531584 14255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.532013 14258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.531984 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.532377 14145 server_base.cc:1034] running on GCE node
I20260926 12:28:52.532510 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.532756 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.533902 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790425732533848 us; error 70 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.535554 14145 init.cc:377] Logged in from keytab as oryx/127.7.99.61@KRBTEST.COM (short username oryx)
I20260926 12:28:52.536020 14145 webserver.cc:460] Webserver started at http://127.7.99.61:42519/ using document root <none> and password file <none>
I20260926 12:28:52.536260 14145 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.536312 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.537636 14145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.538692 14265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.538975 14145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.539054 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "da48ca6921b84758962631c86bf3ceaa"
server_key_iv: "07fb6ecd2084e5f082a9844a89bed11c"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.539134 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.553206 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:52.555372 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:52.560437 14145 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.61:44375
I20260926 12:28:52.560561 14317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.61:44375 every 8 connection(s)
I20260926 12:28:52.561048 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
I20260926 12:28:52.563026 14318 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:28:52.564111 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Bootstrap starting.
I20260926 12:28:52.575048 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:28:52.575513 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: Bootstrap complete.
I20260926 12:28:52.576216 14318 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } }
I20260926 12:28:52.576473 14318 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d18ca25f6ac448fa8e21839e6a4ba216, State: Initialized, Role: LEARNER
I20260926 12:28:52.576638 14318 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } }
I20260926 12:28:52.577068 14321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } }
I20260926 12:28:52.577188 14321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:28:52.577332 14318 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:52.580462 14332 catalog_manager.cc:1259] Loaded cluster ID: 87bcdefce27148d6aacf059caa76cbbf
I20260926 12:28:52.580516 14332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: loading cluster ID for follower catalog manager: success
I20260926 12:28:52.582695 14332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: acquiring CA information for follower catalog manager: success
I20260926 12:28:52.584141 14332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:28:52.636049 14254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:52.636368 14254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:52.636449 14254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:52.640538 14254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:52.640583 14254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:52.640605 14254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:52.640621 14254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:52.640635 14254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:52.645533 14254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.60.keytab
--principal=oryx/127.7.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:52.646503 14254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:52.647468 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.655630 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.655624 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.655718 14254 server_base.cc:1034] running on GCE node
W20260926 12:28:52.655639 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.656569 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.657627 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.658776 14254 hybrid_clock.cc:648] HybridClock initialized: now 1790425732658746 us; error 42 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.661779 14254 init.cc:377] Logged in from keytab as oryx/127.7.99.60@KRBTEST.COM (short username oryx)
I20260926 12:28:52.662599 14254 webserver.cc:460] Webserver started at http://127.7.99.60:41413/ using document root <none> and password file <none>
I20260926 12:28:52.663079 14254 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.663132 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.663286 14254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:52.665300 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/instance:
uuid: "b9d1d108102c402d8749c233786a396e"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "32dc2cd24cbf9093faf3a7cba4baeef1"
server_key_iv: "b9e89f5faef4d069febfbb9a5fbe635d"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.665793 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal/instance:
uuid: "b9d1d108102c402d8749c233786a396e"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "32dc2cd24cbf9093faf3a7cba4baeef1"
server_key_iv: "b9e89f5faef4d069febfbb9a5fbe635d"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.669289 14254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:52.671630 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.672832 14254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:52.672963 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "b9d1d108102c402d8749c233786a396e"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "32dc2cd24cbf9093faf3a7cba4baeef1"
server_key_iv: "b9e89f5faef4d069febfbb9a5fbe635d"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.673067 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.691558 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:52.694416 14254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:52.694604 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:52.701663 14254 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.60:34453
I20260926 12:28:52.701922 14399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.60:34453 every 8 connection(s)
I20260926 12:28:52.702597 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
I20260926 12:28:52.705379 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:52.710698 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14254
I20260926 12:28:52.710356 14400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.60@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
I20260926 12:28:52.724905 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.713403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.7.99.62:43561)
0926 12:28:52.713523 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.713527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.713802 (+ 275us) server_negotiation.cc:407] Connection header received
0926 12:28:52.714217 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.714220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.714273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.714354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.715366 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.716154 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.717596 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.717923 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.718035 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.720647 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.720651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.720654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.720682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.722467 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.723041 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.723045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.723047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.723131 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.723519 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.723522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.723524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.723710 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.723770 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.724548 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.724695 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:28:52.725618 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.713280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56678)
0926 12:28:52.713665 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.713709 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.714007 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.714425 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.714431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.714878 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.715230 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.715245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.716302 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.716305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.717465 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.717476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.718069 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.718072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.718174 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.718785 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.718810 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.720446 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.722594 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.722602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.722610 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.722940 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.723228 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.723231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.723233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.723397 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.723840 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.723846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.724266 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.724502 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.724900 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":129,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260926 12:28:52.725621 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.712606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.7.99.60:34453)
0926 12:28:52.713121 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.713141 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.713179 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:28:52.713964 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.713982 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.714200 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.714390 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.715245 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.716468 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.717595 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.718141 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.718220 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.721133 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.721138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.721146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.721166 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.722813 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.723302 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.723306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.723307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.723389 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.723679 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.723682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.723683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.724049 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.724264 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.724502 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.724816 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":379,"spinlock_wait_cycles":6016,"thread_start_us":104,"threads_started":1}
I20260926 12:28:52.727154 14400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.60@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
I20260926 12:28:52.734879 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.727663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:60004)
0926 12:28:52.727786 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.727799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.727869 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.728282 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.728285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.728453 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.728532 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.728537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.729406 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.729412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.729942 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.729949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.730395 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.730399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.730498 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.730891 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.730902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.731968 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.733480 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.733485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.733487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.733666 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.733941 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.733964 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.733966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.734050 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.734402 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.734405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.734490 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.734599 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.734715 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 12:28:52.735124 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.727764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.7.99.61:44375)
0926 12:28:52.728036 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.728039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.728051 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:52.728083 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.728085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.728127 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.728272 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.728641 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.729300 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.730057 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.730285 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.730399 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.732104 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.732107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.732109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.732131 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.733379 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.733759 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.733764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.733766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.733846 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.734143 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.734147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.734148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.734311 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.734411 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.734646 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.734797 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260926 12:28:52.735754 14400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.60@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
I20260926 12:28:52.743018 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.736144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.0.0.1:56194)
0926 12:28:52.736277 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.736289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.736365 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.736542 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.736544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.736683 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.736758 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.736770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.737385 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.737387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.737883 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.737889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.738693 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.738696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.738763 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.739089 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.739098 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.740259 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.741873 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.741876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.741877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.742044 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.742266 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.742268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.742269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.742339 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.742649 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.742651 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.742713 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.742799 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.742895 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 12:28:52.743084 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.736202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.7.99.60:34453)
0926 12:28:52.736316 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.736320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.736339 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:28:52.736435 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.736437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.736472 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.736527 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.736851 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.737306 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.737973 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.738570 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.738643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.740382 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.740386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.740394 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.740414 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.741781 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.742122 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.742124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.742125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.742187 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.742408 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.742410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.742424 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.742575 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.742644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.742825 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.742940 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":38}
I20260926 12:28:52.745712 14400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Bootstrap starting.
I20260926 12:28:52.748169 14400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:52.748950 14400 log.cc:824] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:52.750869 14400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: No bootstrap required, opened a new log
I20260926 12:28:52.753758 14400 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } }
I20260926 12:28:52.753957 14400 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:52.754032 14400 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d1d108102c402d8749c233786a396e, State: Initialized, Role: FOLLOWER
I20260926 12:28:52.754415 14400 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } }
I20260926 12:28:52.754835 14406 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } } }
I20260926 12:28:52.754969 14406 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:52.755523 14400 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:28:52.759954 14417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:28:52.760053 14417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:28:52.763120 14254 master_runner.cc:428] Detected that this master b9d1d108102c402d8749c233786a396e is joining an existing cluster
I20260926 12:28:52.763165 14254 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.99.60:34453
I20260926 12:28:52.770673 13872 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
I20260926 12:28:52.773984 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.7.99.61:44375)
0926 12:28:52.765319 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.765324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.766377 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 12:28:52.766527 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.766530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.766575 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.766622 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.767203 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.767985 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769233 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769577 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769650 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.770941 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.770945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.770948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.770973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.772523 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.772968 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.772970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.772972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.773039 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.773308 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.773310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.773311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.773469 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.773538 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.773783 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.773873 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:28:52.774151 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:60014)
0926 12:28:52.766310 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.766321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.766429 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.766747 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.766750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.766966 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.767071 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.767077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.768116 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.768119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.769102 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.769112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769791 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.769913 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.770333 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.770348 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.770802 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.772621 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.772625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.772627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.772877 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.773123 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.773126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.773128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.773223 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.773555 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.773558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.773645 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.773755 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.773887 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":27,"thread_start_us":81,"threads_started":1}
I20260926 12:28:52.778967 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13743
I20260926 12:28:52.785272 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:38113
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:43561
--webserver_port=33803
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:28:52.801038 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.0.0.1:56204)
0926 12:28:52.766025 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.766037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.766128 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.766396 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.766401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.766625 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.766729 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.766736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.767609 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.767612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.768845 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.768852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769405 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.769853 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.796806 (+ 26953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.796827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.797323 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.799359 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.799364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.799368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.799635 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.799911 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.799914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.799919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.799992 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.800495 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.800497 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.800582 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.800674 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.800822 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":24,"thread_start_us":69,"threads_started":1}
I20260926 12:28:52.801350 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.7.99.62:43561)
0926 12:28:52.765555 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.765559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.765778 (+ 219us) server_negotiation.cc:407] Connection header received
0926 12:28:52.765901 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.765903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.765940 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.766033 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.766554 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.767393 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.768687 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769052 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769117 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.797359 (+ 28242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.797367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.797373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.797410 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.799591 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.800115 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.800118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.800120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.800215 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.800490 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.800493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.800495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.800695 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.800754 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.801078 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.801193 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:52.801960 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.7.99.60:34453)
0926 12:28:52.765318 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.765323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.766096 (+ 773us) server_negotiation.cc:407] Connection header received
0926 12:28:52.766205 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.766209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.766247 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.766320 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.766832 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.767509 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.768940 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769233 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769295 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.797492 (+ 28197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.797496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.797499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.797524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.799241 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.799733 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.799738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.799742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.799820 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.800086 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.800089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.800090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.800367 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.800451 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.801688 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.801817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:28:52.802234 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.765075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56690)
0926 12:28:52.765692 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.765709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.765806 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.766048 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.766051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.766289 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.766421 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.766430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.767515 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.767518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.768533 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.768545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.769260 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.769264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.769361 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.769880 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.769909 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.797135 (+ 27226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.799762 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.799764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.799769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.799998 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.800320 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.800324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.800325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.800395 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.800820 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.800823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.800893 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.801670 (+ 777us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.801876 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":86,"spinlock_wait_cycles":4352,"thread_start_us":95,"threads_started":1}
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425731, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.62@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
I20260926 12:28:52.813323 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.805820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.7.99.60:34453)
0926 12:28:52.805912 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.805916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.805966 (+ 50us) server_negotiation.cc:407] Connection header received
0926 12:28:52.806126 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.806129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.806183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.806262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.806759 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.807272 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.808096 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.808382 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.808490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.810400 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.810403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.810406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.810427 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.811793 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.812222 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.812227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.812229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.812301 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.812568 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.812574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.812575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.812723 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.812824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.813078 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.813172 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 12:28:52.813323 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.805771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.0.0.1:56210)
0926 12:28:52.805899 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.805915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.806007 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.806373 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.806376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.806541 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.806638 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.806645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.807380 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.807385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.807984 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.807992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.808490 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.808494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.808587 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.809018 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.809031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.810276 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.811901 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.811907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.811909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.812101 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.812391 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.812395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.812397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.812481 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.812815 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.812818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.812914 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.813022 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.813175 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 12:28:52.814503 13681 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.99.60:34453: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b9d1d108102c402d8749c233786a396e"
member_type: NON_VOTER
last_known_addr {
host: "127.7.99.60"
port: 34453
}
attrs {
promote: true
}
}
dest_uuid: "08ec1828ecc2448ca760196b3cd9599e"
cas_config_opid_index: 5
I20260926 12:28:52.815055 13681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } }
I20260926 12:28:52.816087 14240 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b9d1d108102c402d8749c233786a396e (127.7.99.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.816731 14374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:52.817538 14240 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.99.60:34453
I20260926 12:28:52.817704 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.817790 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:52.818032 14374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:28:52.818596 14254 master.cc:537] Master@127.7.99.60:34453 shutting down...
I20260926 12:28:52.819363 14241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:28:52.821488 13667 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e -> Peer b9d1d108102c402d8749c233786a396e (127.7.99.60:34453): Couldn't send request to peer b9d1d108102c402d8749c233786a396e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:28:52.827025 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.816359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.7.99.61:44375)
0926 12:28:52.816484 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.816488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.821706 (+ 5218us) server_negotiation.cc:407] Connection header received
0926 12:28:52.821895 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.821899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.821973 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.822036 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:52.822376 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.823050 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.824769 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.825375 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.825504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.825670 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.826710 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.826850 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:28:52.827239 14254 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:52.827541 14254 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:52.827658 14254 tablet_replica.cc:330] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: stopping tablet replica
I20260926 12:28:52.827762 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.816565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:60024)
0926 12:28:52.821633 (+ 5068us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.821648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.821776 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.822182 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.822185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.822199 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:52.822275 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.822281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.823176 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.823179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.824645 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.824655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.826210 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.826216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.826392 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.826501 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.827437 (+ 936us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.827543 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4998}
I20260926 12:28:52.828445 14284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 LEARNER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:28:52.828891 14241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 12:28:52.830698 14284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b9d1d108102c402d8749c233786a396e (127.7.99.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.833102 14241 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: attempting to promote NON_VOTER d18ca25f6ac448fa8e21839e6a4ba216 to VOTER
I20260926 12:28:52.833065 14321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec1828ecc2448ca760196b3cd9599e. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.833170 14321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:28:52.833251 14321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: NON_VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: true } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.833314 14321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:28:52.833549 14241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } }
W20260926 12:28:52.835259 13667 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e -> Peer b9d1d108102c402d8749c233786a396e (127.7.99.60:34453): Couldn't send request to peer b9d1d108102c402d8749c233786a396e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:28:52.835681 14284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 LEARNER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 12:28:52.836282 14242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 12:28:52.837841 14240 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.839136 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.839246 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:52.839383 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.839514 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:52.840046 14284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.841358 14428 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.841475 14428 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:52.847842 14254 master.cc:552] Master@127.7.99.60:34453 shutdown complete.
I20260926 12:28:52.847913 14254 master_runner.cc:305] Clearing existing system tablet
I20260926 12:28:52.848143 14254 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.848240 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.849314 14254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.850159 14430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.850387 14254 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.850514 14254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.850565 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "b9d1d108102c402d8749c233786a396e"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "32dc2cd24cbf9093faf3a7cba4baeef1"
server_key_iv: "b9e89f5faef4d069febfbb9a5fbe635d"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.852423 14254 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:52.854411 14254 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:28:52.854542 14254 log.cc:1196] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 12:28:52.854801 14254 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Deleting consensus metadata
I20260926 12:28:52.855000 14254 master_runner.cc:315] Copying system tablet from 127.7.99.62:43561
I20260926 12:28:52.856134 14254 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: tablet copy: Beginning tablet copy session from remote peer at address 127.7.99.62:43561
I20260926 12:28:52.867908 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.856826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.7.99.62:43561)
0926 12:28:52.857001 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:52.857007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:52.859842 (+ 2835us) server_negotiation.cc:407] Connection header received
0926 12:28:52.860015 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:52.860018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:52.860073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:52.860165 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:52.860680 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.861449 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.862957 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.863264 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.863387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.864752 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:52.864756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:52.864758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:52.864782 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:52.866278 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.866717 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.866721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.866723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.866807 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:52.867078 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:52.867083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:52.867085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:52.867262 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:52.867366 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:52.867634 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:28:52.867744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:28:52.867923 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:52.856758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:56706)
0926 12:28:52.859737 (+ 2979us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:52.859750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:52.859894 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:52.860249 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:52.860252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:52.860448 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:52.860559 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.860565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.861569 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.861575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.862783 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:52.862793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:52.863420 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:52.863424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:52.863594 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:52.864257 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:52.864275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:52.864615 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:52.866396 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.866402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.866404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.866615 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.866898 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:52.866902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:52.866904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:52.866989 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:52.867355 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:52.867358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:52.867460 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 12:28:52.867582 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 12:28:52.867730 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2917,"thread_start_us":109,"threads_started":1}
I20260926 12:28:52.868692 13701 tablet_copy_service.cc:140] P 08ec1828ecc2448ca760196b3cd9599e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b9d1d108102c402d8749c233786a396e ({username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:56706)
I20260926 12:28:52.868808 13701 tablet_copy_service.cc:161] P 08ec1828ecc2448ca760196b3cd9599e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b9d1d108102c402d8749c233786a396e at {username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:56706: session id = b9d1d108102c402d8749c233786a396e-00000000000000000000000000000000
I20260926 12:28:52.869656 13701 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:28:52.870786 14254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:28:52.872725 14254 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: tablet copy: Starting download of 0 data blocks...
I20260926 12:28:52.872926 14254 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: tablet copy: Starting download of 1 WAL segments...
I20260926 12:28:52.874262 14254 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:28:52.875787 13701 tablet_copy_service.cc:342] P 08ec1828ecc2448ca760196b3cd9599e: Request end of tablet copy session b9d1d108102c402d8749c233786a396e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:56706
I20260926 12:28:52.875885 13701 tablet_copy_service.cc:433] P 08ec1828ecc2448ca760196b3cd9599e: ending tablet copy session b9d1d108102c402d8749c233786a396e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b9d1d108102c402d8749c233786a396e
I20260926 12:28:52.882804 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.884403 14437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.884617 14440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.884886 14254 server_base.cc:1034] running on GCE node
W20260926 12:28:52.888322 14438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.888577 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.888916 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.890054 14254 hybrid_clock.cc:648] HybridClock initialized: now 1790425732890067 us; error 46 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.891635 14254 init.cc:377] Logged in from keytab as oryx/127.7.99.60@KRBTEST.COM (short username oryx)
I20260926 12:28:52.892145 14254 webserver.cc:460] Webserver started at http://127.7.99.60:33391/ using document root <none> and password file <none>
I20260926 12:28:52.892374 14254 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.892433 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.893563 14254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.894510 14447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.894731 14254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:52.894827 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "b9d1d108102c402d8749c233786a396e"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "32dc2cd24cbf9093faf3a7cba4baeef1"
server_key_iv: "b9e89f5faef4d069febfbb9a5fbe635d"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.894935 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.902192 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:52.904266 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:52.909144 14254 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.60:34453
I20260926 12:28:52.909160 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.60:34453 every 8 connection(s)
I20260926 12:28:52.909710 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
I20260926 12:28:52.911146 14500 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:28:52.912060 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Bootstrap starting.
I20260926 12:28:52.920945 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:28:52.921241 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Bootstrap complete.
I20260926 12:28:52.921821 14500 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } }
I20260926 12:28:52.921945 14500 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9d1d108102c402d8749c233786a396e, State: Initialized, Role: LEARNER
I20260926 12:28:52.922055 14500 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } }
I20260926 12:28:52.922354 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:52.922510 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: LEARNER
I20260926 12:28:52.922518 14500 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:52.926162 14514 catalog_manager.cc:1259] Loaded cluster ID: 87bcdefce27148d6aacf059caa76cbbf
I20260926 12:28:52.926237 14514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: loading cluster ID for follower catalog manager: success
I20260926 12:28:52.927690 14514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: acquiring CA information for follower catalog manager: success
I20260926 12:28:52.928843 14514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:28:52.941727 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:52.942049 14426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:52.942108 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:52.948235 14426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:52.948318 14426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:52.948417 14426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:52.956338 14426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:38113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=33803
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:52.957540 14426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:52.958750 14426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:52.969344 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.969383 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:52.969347 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:52.969560 14426 server_base.cc:1034] running on GCE node
I20260926 12:28:52.970012 14426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:52.970654 14426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:52.971807 14426 hybrid_clock.cc:648] HybridClock initialized: now 1790425732971796 us; error 28 us; skew 500 ppm
Sep 26 12:28:52 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:52.974530 14426 init.cc:377] Logged in from keytab as oryx/127.7.99.1@KRBTEST.COM (short username oryx)
I20260926 12:28:52.975322 14426 webserver.cc:460] Webserver started at http://127.7.99.1:33803/ using document root <none> and password file <none>
I20260926 12:28:52.975845 14426 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:52.975944 14426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:52.979877 14426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:28:52.981841 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:52.982700 14006 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
I20260926 12:28:52.983081 14426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:52.983186 14426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "79ed588c4ac14303bd5c9c25d8b5afe5"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "e1574881cedc8ee3b810f671150134bb"
server_key_iv: "155fea2865522285e5ea4b6329d64798"
server_key_version: "encryptionkey@0"
I20260926 12:28:52.983790 14426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:52.997406 14426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:53.000458 14426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:53.000628 14426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:53.001176 14426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:53.001941 14426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:53.002013 14426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.002077 14426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:53.002126 14426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.010443 14426 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:38113
I20260926 12:28:53.010486 14642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:38113 every 8 connection(s)
I20260926 12:28:53.011386 14426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:53.013499 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14426
I20260926 12:28:53.013700 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13877
I20260926 12:28:53.021416 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:44669
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:43561
--webserver_port=38625
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425732, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:53.029907 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42015 (local address 127.7.99.61:44375)
0926 12:28:53.013922 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.013929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.015789 (+ 1860us) server_negotiation.cc:407] Connection header received
0926 12:28:53.015932 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.015936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.015998 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.016053 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.018052 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.018835 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.020791 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021133 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021206 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.025708 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.025714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.025720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.025754 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.027553 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.028140 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.028147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.028151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.028372 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.028756 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.028763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.028767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.029035 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.029131 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.029526 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.029680 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:28:53.030174 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49911 (local address 127.7.99.62:43561)
0926 12:28:53.013223 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.013227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.014880 (+ 1653us) server_negotiation.cc:407] Connection header received
0926 12:28:53.015932 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.015936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.015998 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.016053 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.018052 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.018835 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.020784 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021136 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021206 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.026364 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.026368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.026371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.026400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.028410 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.028927 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.028930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.028932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.029026 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.029316 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.029319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.029321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.029509 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.029589 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.029896 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.030032 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:28:53.030586 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:54049 (local address 127.7.99.60:34453)
0926 12:28:53.014682 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.014687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.016082 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 12:28:53.016296 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.016302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.016364 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.016430 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.018047 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.019268 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.020798 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021175 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021258 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.024830 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.024833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.024836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.024868 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.026751 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.027398 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.027404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.027409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.027524 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.027975 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.027981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.027983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.028233 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.028331 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.030272 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.030394 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1432,"thread_start_us":84,"threads_started":1}
I20260926 12:28:53.030920 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:49911)
0926 12:28:53.014715 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.014787 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.015692 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.016453 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.016461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.017083 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.017876 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.017906 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.019028 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.019031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.020645 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.020656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021407 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.021745 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.022583 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.022603 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.025687 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.028525 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.028527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.028529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.028812 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.029132 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.029135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.029137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.029217 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.029651 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.029654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.029731 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.030326 (+ 595us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.030456 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":419,"spinlock_wait_cycles":18688,"thread_start_us":118,"threads_started":1}
I20260926 12:28:53.031204 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.7.99.1:42015)
0926 12:28:53.014868 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.014881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.015692 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.016461 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.016465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.017107 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.017876 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.017906 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.019027 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.019031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.020677 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.020686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021407 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.021742 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.022683 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.022701 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.025536 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.027709 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.027715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.027719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.028012 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.028513 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.028518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.028522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.028637 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.029199 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.029202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.029312 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.030137 (+ 825us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.030312 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":541,"spinlock_wait_cycles":28800,"thread_start_us":127,"threads_started":1}
I20260926 12:28:53.031621 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.013510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.7.99.1:54049)
0926 12:28:53.015992 (+ 2482us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.016008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.016154 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.016612 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.016618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.017103 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.017878 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.017909 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.019413 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.019419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.020678 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.020689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.021443 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.021447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.021738 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.022354 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.022380 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.024676 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.026888 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.026897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.026914 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.027281 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.027658 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.027664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.027669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.027857 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.028384 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.028391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.028709 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.028895 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.029225 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2378,"mutex_wait_us":1079,"thread_start_us":119,"threads_started":1}
I20260926 12:28:53.032128 14645 heartbeater.cc:347] Connected to a master server at 127.7.99.61:44375
I20260926 12:28:53.032128 14644 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:53.032392 14644 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.032394 14645 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.032979 14645 heartbeater.cc:511] Master 127.7.99.61:44375 requested a full tablet report, sending...
I20260926 12:28:53.033033 14643 heartbeater.cc:347] Connected to a master server at 127.7.99.60:34453
I20260926 12:28:53.033043 14644 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:53.033131 14643 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.033320 14643 heartbeater.cc:511] Master 127.7.99.60:34453 requested a full tablet report, sending...
I20260926 12:28:53.034128 13681 ts_manager.cc:194] Re-registered known tserver with Master: 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
I20260926 12:28:53.034416 14464 ts_manager.cc:194] Registered new tserver with Master: 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
I20260926 12:28:53.034417 14274 ts_manager.cc:194] Registered new tserver with Master: 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
I20260926 12:28:53.034839 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:49911
W20260926 12:28:53.134132 14651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:53.134387 14651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:53.134457 14651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:53.138427 14651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:53.138507 14651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:53.138589 14651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:53.144516 14651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:44669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=38625
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:53.145670 14651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:53.146677 14651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:53.155586 14656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:53.155604 14657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:53.155754 14651 server_base.cc:1034] running on GCE node
W20260926 12:28:53.155881 14659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:53.156297 14651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:53.156862 14651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:53.158030 14651 hybrid_clock.cc:648] HybridClock initialized: now 1790425733158005 us; error 46 us; skew 500 ppm
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:53.160775 14651 init.cc:377] Logged in from keytab as oryx/127.7.99.2@KRBTEST.COM (short username oryx)
I20260926 12:28:53.161623 14651 webserver.cc:460] Webserver started at http://127.7.99.2:38625/ using document root <none> and password file <none>
I20260926 12:28:53.162196 14651 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:53.162297 14651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:53.166188 14651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:53.168147 14666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.169400 14651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:53.169533 14651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "32fdeb060b434521b8aa5571c0e82196"
format_stamp: "Formatted at 2026-09-26 12:28:51 on dist-test-slave-3jw2"
server_key: "ec16891ea3d482d286879eb9b4227168"
server_key_iv: "3f64a09a5de5cd5167e6cb695ce75dc2"
server_key_version: "encryptionkey@0"
I20260926 12:28:53.170051 14651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:53.179246 14651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:53.182272 14651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:53.182461 14651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:53.183009 14651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:53.183802 14651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:53.183847 14651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.183914 14651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:53.183944 14651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.192704 14651 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:44669
I20260926 12:28:53.192761 14779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:44669 every 8 connection(s)
I20260926 12:28:53.193706 14651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:53.197317 14140 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
I20260926 12:28:53.198477 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14651
I20260926 12:28:53.198647 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14011
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
I20260926 12:28:53.207401 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:34519
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:43561
--webserver_port=45823
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:28:53.208792 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.196323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:48003 (local address 127.7.99.61:44375)
0926 12:28:53.196441 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.196446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.197357 (+ 911us) server_negotiation.cc:407] Connection header received
0926 12:28:53.198010 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.198014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.198068 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.198119 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.199758 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.200390 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.201552 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.201777 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.201885 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.204408 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.204412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.204415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.204455 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.206197 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.206859 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.206865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.206870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.206970 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.207367 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.207370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.207372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.207609 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.207683 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.208419 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.208552 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:53.209824 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.195450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:41021 (local address 127.7.99.60:34453)
0926 12:28:53.195601 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.195604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.197038 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 12:28:53.198029 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.198032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.198080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.198129 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.199769 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.200699 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.202090 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.202511 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.202599 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.205336 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.205340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.205342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.205369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.207119 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.208033 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.208038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.208040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.208121 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.208480 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.208484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.208486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.208725 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.208800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.209507 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.209637 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:28:53.210430 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.196177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.7.99.2:48003)
0926 12:28:53.197258 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.197278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.197800 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.198529 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.198538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.199075 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.199620 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.199633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.200541 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.200543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.201414 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.201422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.201916 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.201920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.202076 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.202629 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.202644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.204209 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.206363 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.206371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.206389 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.206748 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.207089 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.207092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.207094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.207261 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.207765 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.207775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.208083 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.208973 (+ 890us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.209364 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":29,"thread_start_us":100,"threads_started":1}
I20260926 12:28:53.211294 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.195915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.7.99.2:41021)
0926 12:28:53.196937 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.196948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.197783 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.198530 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.198539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.199250 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.199622 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.199637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.200857 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.200863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.201959 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.201973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.202756 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.202762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.202883 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.203401 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.203420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.205192 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.207374 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.207378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.207382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.207888 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.208273 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.208276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.208278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.208369 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.209154 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.209157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.209315 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.210298 (+ 983us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.210444 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":67,"spinlock_wait_cycles":47488,"thread_start_us":180,"threads_started":1}
I20260926 12:28:53.212085 14780 heartbeater.cc:347] Connected to a master server at 127.7.99.60:34453
I20260926 12:28:53.212141 14782 heartbeater.cc:347] Connected to a master server at 127.7.99.61:44375
I20260926 12:28:53.212401 14780 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.212400 14782 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.213042 14780 heartbeater.cc:511] Master 127.7.99.60:34453 requested a full tablet report, sending...
I20260926 12:28:53.213043 14782 heartbeater.cc:511] Master 127.7.99.61:44375 requested a full tablet report, sending...
I20260926 12:28:53.213438 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.195458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:34679 (local address 127.7.99.62:43561)
0926 12:28:53.195677 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.195680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.196817 (+ 1137us) server_negotiation.cc:407] Connection header received
0926 12:28:53.198019 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.198023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.198075 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.198129 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.204796 (+ 6667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.205565 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.206786 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.207154 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.207226 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.209808 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.209812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.209814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.209842 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.211589 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.212072 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.212077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.212080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.212192 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.212518 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.212522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.212523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.212721 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.212809 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.213188 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.213302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260926 12:28:53.213846 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.195915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:34679)
0926 12:28:53.196680 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.196727 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.197783 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.204295 (+ 6512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.204301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.204498 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.204640 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.204649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.205694 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.205700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.206643 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.206657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.207391 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.207394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.207515 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.207976 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.207992 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.209666 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.211718 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.211720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.211723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.211965 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.212309 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.212312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.212314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.212397 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.212902 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.212904 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.213001 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.213563 (+ 562us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.213699 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"spinlock_wait_cycles":13696,"thread_start_us":161,"threads_started":1}
I20260926 12:28:53.214329 14274 ts_manager.cc:194] Registered new tserver with Master: 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:53.215281 14781 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:53.215391 14781 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.215389 14464 ts_manager.cc:194] Registered new tserver with Master: 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:53.215986 14781 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:53.216674 13681 ts_manager.cc:194] Re-registered known tserver with Master: 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:53.217377 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:34679
I20260926 12:28:53.218962 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.214590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.7.99.60:34453)
0926 12:28:53.214732 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.214739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.214753 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:28:53.214797 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.214802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.214867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.214978 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.215375 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.216383 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.217763 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.218382 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.218501 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.218637 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.218798 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.218862 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:53.219051 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.214153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.0.0.1:56226)
0926 12:28:53.214244 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.214260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.214343 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.215105 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.215111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.215134 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.215242 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.215251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.216504 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.216507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.217648 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.217656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.218523 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.218528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.218653 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.218688 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.218778 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.218872 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 12:28:53.219615 14474 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 12:28:53.219902 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec1828ecc2448ca760196b3cd9599e. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:53.220033 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: LEARNER
W20260926 12:28:53.318933 14787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:53.319167 14787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:53.319207 14787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:53.323287 14787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:53.323335 14787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:53.323417 14787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:53.328542 14787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:34519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=45823
--tserver_master_addrs=127.7.99.62:43561,127.7.99.61:44375,127.7.99.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:53.329510 14787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:53.330495 14787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:53.340225 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:53.340278 14795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:53.340267 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:53.340561 14787 server_base.cc:1034] running on GCE node
I20260926 12:28:53.341022 14787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:53.341647 14787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:53.342861 14787 hybrid_clock.cc:648] HybridClock initialized: now 1790425733342841 us; error 29 us; skew 500 ppm
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:53.345890 14787 init.cc:377] Logged in from keytab as oryx/127.7.99.3@KRBTEST.COM (short username oryx)
I20260926 12:28:53.346700 14787 webserver.cc:460] Webserver started at http://127.7.99.3:45823/ using document root <none> and password file <none>
I20260926 12:28:53.347203 14787 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:53.347276 14787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:53.351095 14787 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:53.353144 14802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.354403 14787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:53.354506 14787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "b8e1ac7a94414f268bc12640cdccba09"
format_stamp: "Formatted at 2026-09-26 12:28:52 on dist-test-slave-3jw2"
server_key: "7d5ac97a3cc128bcb36deca4cec95bc8"
server_key_iv: "d74d1b4ac0ae46571bc10a71e817cbb0"
server_key_version: "encryptionkey@0"
I20260926 12:28:53.355023 14787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:53.366508 14787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:53.369735 14787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:53.369890 14787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:53.370401 14787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:28:53.371212 14787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:53.371261 14787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.371313 14787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:53.371340 14787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:53.381626 14787 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:34519
I20260926 12:28:53.381644 14915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:34519 every 8 connection(s)
I20260926 12:28:53.382688 14787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:53.384266 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 14787
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.60@KRBTEST.COM
Sep 26 12:28:53 dist-test-slave-3jw2 krb5kdc[13636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425733, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.61@KRBTEST.COM
I20260926 12:28:53.402619 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.387802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.7.99.61:44375)
0926 12:28:53.388131 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.388135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.389322 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 12:28:53.389372 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.389377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.389439 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.389541 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.390470 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.391270 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.392522 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.393703 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.396881 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.396885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.396887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.396915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.400341 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.400847 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.400854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.400859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.400946 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.401249 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.401252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.401254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.401496 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.401628 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.402289 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.402411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":260,"thread_start_us":81,"threads_started":1}
I20260926 12:28:53.404516 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.384811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:38143 (local address 127.7.99.60:34453)
0926 12:28:53.384945 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.384950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.386469 (+ 1519us) server_negotiation.cc:407] Connection header received
0926 12:28:53.388482 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.388487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.388548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.388644 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.391302 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.392056 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393815 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.394116 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.394186 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.399097 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.399100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.399103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.399128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.400976 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.401570 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.401574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.401575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.401656 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403209 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.403213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.403214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.403389 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.403489 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.404254 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.404367 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:53.405192 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.386186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35803 (local address 127.7.99.62:43561)
0926 12:28:53.386331 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.386338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.386562 (+ 224us) server_negotiation.cc:407] Connection header received
0926 12:28:53.387764 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.387767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.387820 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.387896 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.389978 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.390821 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393021 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.393351 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393439 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.397848 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.397852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.397855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.397881 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.400727 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.402009 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.402013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.402014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.402096 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.402469 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.402472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.402474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.402664 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.403778 (+ 1114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.404955 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.405061 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:28:53.405476 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.385125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:35803)
0926 12:28:53.386292 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.386339 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.387420 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.388248 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.388256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.388871 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.389840 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.389848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.390993 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.390996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.392896 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.392903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.393544 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.393690 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.394364 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.394388 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.397025 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.401183 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.401187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.401191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.401899 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.402196 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.402199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.402200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.402365 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.402796 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.402802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.403855 (+ 1053us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.404009 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.404384 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":1799,"spinlock_wait_cycles":34688,"thread_start_us":83,"threads_started":1}
I20260926 12:28:53.405411 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.387528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.7.99.62:43561)
0926 12:28:53.388296 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.388301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.389065 (+ 764us) server_negotiation.cc:407] Connection header received
0926 12:28:53.389107 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.389109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.389160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.389239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.390381 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.391230 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.395645 (+ 4415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.396012 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.396114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.399096 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.399100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.399103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.399128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.403047 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403556 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.403559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.403561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.403643 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403971 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.403975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.403977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.404237 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.404379 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.405065 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.405204 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":95,"threads_started":1}
I20260926 12:28:53.405884 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.385279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35307 (local address 127.7.99.61:44375)
0926 12:28:53.385483 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.385488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.386519 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 12:28:53.387679 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.387686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.387750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.387813 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.390152 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.390905 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.392560 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.392987 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393095 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.399281 (+ 6186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.399287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.399303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.399333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.402147 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403306 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.403310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.403314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.403401 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.404216 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.404220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.404222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.404434 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.404519 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.405596 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.405716 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":116}
I20260926 12:28:53.406498 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.387073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.7.99.60:34453)
0926 12:28:53.387416 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.387421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.387992 (+ 571us) server_negotiation.cc:407] Connection header received
0926 12:28:53.388878 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.388882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.388931 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.388997 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.391420 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.392238 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.396383 (+ 4145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.396718 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.396850 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.399286 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.399291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.399295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.399321 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.402426 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403767 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.403770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.403772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.403863 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.404595 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.404601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.404606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.404842 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.404953 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.406166 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.406303 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260926 12:28:53.407073 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.385125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.7.99.3:35307)
0926 12:28:53.386282 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.386338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.387423 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.388245 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.388255 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.388917 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.390031 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.390039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.391028 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.391032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.392426 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.392435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.393269 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.393274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.393485 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.395450 (+ 1965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.395467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.399108 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.402560 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.402565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.402567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.402824 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.403545 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.403550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.403643 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.405020 (+ 1377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.405023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.405126 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.406678 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.406832 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":835,"spinlock_wait_cycles":281472,"thread_start_us":138,"threads_started":1}
I20260926 12:28:53.407410 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.385217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.7.99.3:38143)
0926 12:28:53.386282 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.386338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.388347 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.388774 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.388780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.389255 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:53.389630 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.389655 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.392878 (+ 3223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.392881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.393692 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.393699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.394347 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.394350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.394449 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.395201 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:53.395215 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:53.398945 (+ 3730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:53.401091 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.401101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.401118 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.401456 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.403023 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:53.403026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:53.403028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:53.403107 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:53.403554 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:53.403557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:53.403786 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.404009 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.406445 (+ 2436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":1370,"spinlock_wait_cycles":88576,"thread_start_us":168,"threads_started":1}
I20260926 12:28:53.407241 14917 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43561
I20260926 12:28:53.407787 14918 heartbeater.cc:347] Connected to a master server at 127.7.99.61:44375
I20260926 12:28:53.407881 14918 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.407883 14917 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.408438 14918 heartbeater.cc:511] Master 127.7.99.61:44375 requested a full tablet report, sending...
I20260926 12:28:53.408439 14917 heartbeater.cc:511] Master 127.7.99.62:43561 requested a full tablet report, sending...
I20260926 12:28:53.408725 14916 heartbeater.cc:347] Connected to a master server at 127.7.99.60:34453
I20260926 12:28:53.408815 14916 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:53.408985 14916 heartbeater.cc:511] Master 127.7.99.60:34453 requested a full tablet report, sending...
I20260926 12:28:53.409482 13681 ts_manager.cc:194] Re-registered known tserver with Master: b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519)
I20260926 12:28:53.409525 14274 ts_manager.cc:194] Registered new tserver with Master: b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519)
I20260926 12:28:53.409622 14464 ts_manager.cc:194] Registered new tserver with Master: b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519)
I20260926 12:28:53.410171 13681 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:35803
I20260926 12:28:53.408861 13680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:53.412356 13680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:53.414779 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: NON_VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: true } } }
I20260926 12:28:53.414904 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:53.418479 14934 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: attempting to promote NON_VOTER b9d1d108102c402d8749c233786a396e to VOTER
I20260926 12:28:53.418831 14934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:53.421182 14284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:28:53.421487 14474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 LEARNER]: Refusing update from remote peer 08ec1828ecc2448ca760196b3cd9599e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:28:53.421561 14241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:28:53.421988 14936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:28:53.423619 14241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b9d1d108102c402d8749c233786a396e (127.7.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.424659 14474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b9d1d108102c402d8749c233786a396e (127.7.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.425354 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.425463 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:53.425589 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.425664 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:53.425697 14935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.425844 14935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:53.425962 14284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b9d1d108102c402d8749c233786a396e (127.7.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.426545 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.426662 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:53.427628 14944 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec1828ecc2448ca760196b3cd9599e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:53.427767 14944 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:53.434204 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.429101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:34519 (local address 127.0.0.1:46516)
0926 12:28:53.429263 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.429279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.429375 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.430146 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.430149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.430167 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.430303 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.430320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.431774 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.431780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.432684 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.432693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.433794 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.433800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.433907 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.433931 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.434012 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.434075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 12:28:53.434299 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.429289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.7.99.3:34519)
0926 12:28:53.429597 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.429607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.429627 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:28:53.429724 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.429730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.429974 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.430488 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.430608 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.431641 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.432799 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.433658 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.433775 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.433930 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.434054 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.434155 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260926 12:28:53.435631 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.429987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38113 (local address 127.0.0.1:52726)
0926 12:28:53.430515 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.430528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.430648 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.431233 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.431239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.431258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.431448 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.431463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.432905 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.432909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.433745 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.433758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.435199 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.435203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.435330 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.435357 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.435427 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.435494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":74,"thread_start_us":69,"threads_started":1}
I20260926 12:28:53.435665 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.429681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44669 (local address 127.0.0.1:43450)
0926 12:28:53.430233 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.430244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.430345 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.431293 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.431295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.431307 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.431476 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.431482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.432936 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.432939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.434084 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.434094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.435356 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.435359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.435450 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.435468 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.435509 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.435559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":61,"thread_start_us":98,"threads_started":1}
I20260926 12:28:53.436141 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.430155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.7.99.1:38113)
0926 12:28:53.430490 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.430501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.430625 (+ 124us) server_negotiation.cc:407] Connection header received
0926 12:28:53.430755 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.430760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.430954 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.431094 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.431587 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.432774 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.433893 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.435092 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.435192 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.435312 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.435469 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.435557 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260926 12:28:53.436141 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.429886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.7.99.2:44669)
0926 12:28:53.430881 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.430894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.430914 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:28:53.430992 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.430997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.431160 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.431584 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.431720 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.432835 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.434445 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.435222 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.435816 (+ 594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.435931 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.435970 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.436036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":84,"threads_started":1}
I20260926 12:28:53.438072 14850 tablet_service.cc:1467] Processing CreateTablet for tablet aed52210afb541b384d47a9e1f849440 (DEFAULT_TABLE table=test-table [id=f3b1404e72ba4951a7d0a9b87e1ba1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:53.439092 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed52210afb541b384d47a9e1f849440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:53.442461 14714 tablet_service.cc:1467] Processing CreateTablet for tablet aed52210afb541b384d47a9e1f849440 (DEFAULT_TABLE table=test-table [id=f3b1404e72ba4951a7d0a9b87e1ba1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:53.442798 14577 tablet_service.cc:1467] Processing CreateTablet for tablet aed52210afb541b384d47a9e1f849440 (DEFAULT_TABLE table=test-table [id=f3b1404e72ba4951a7d0a9b87e1ba1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:53.443393 14714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed52210afb541b384d47a9e1f849440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:53.443778 14577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed52210afb541b384d47a9e1f849440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:53.450534 14951 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Bootstrap starting.
I20260926 12:28:53.451267 14950 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Bootstrap starting.
I20260926 12:28:53.451551 14952 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Bootstrap starting.
I20260926 12:28:53.452780 14951 tablet_bootstrap.cc:654] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:53.453197 14950 tablet_bootstrap.cc:654] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:53.453537 14952 tablet_bootstrap.cc:654] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:53.453613 14951 log.cc:824] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:53.453918 14950 log.cc:824] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:53.454332 14952 log.cc:824] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:53.455600 14951 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: No bootstrap required, opened a new log
I20260926 12:28:53.455790 14951 ts_tablet_manager.cc:1397] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:53.456102 14952 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: No bootstrap required, opened a new log
I20260926 12:28:53.456321 14952 ts_tablet_manager.cc:1397] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:53.459245 14951 raft_consensus.cc:348] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.459488 14951 raft_consensus.cc:374] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:53.459548 14951 raft_consensus.cc:729] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fdeb060b434521b8aa5571c0e82196, State: Initialized, Role: FOLLOWER
I20260926 12:28:53.459820 14952 raft_consensus.cc:348] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.459987 14951 consensus_queue.cc:260] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.460045 14952 raft_consensus.cc:374] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:53.460119 14952 raft_consensus.cc:729] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ed588c4ac14303bd5c9c25d8b5afe5, State: Initialized, Role: FOLLOWER
I20260926 12:28:53.460304 14950 tablet_bootstrap.cc:492] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: No bootstrap required, opened a new log
I20260926 12:28:53.460536 14950 ts_tablet_manager.cc:1397] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260926 12:28:53.460559 14952 consensus_queue.cc:260] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.460933 14951 ts_tablet_manager.cc:1428] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:28:53.461131 14781 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
I20260926 12:28:53.461429 14952 ts_tablet_manager.cc:1428] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 12:28:53.462497 14644 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
I20260926 12:28:53.466231 14950 raft_consensus.cc:348] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.466508 14950 raft_consensus.cc:374] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:53.466586 14950 raft_consensus.cc:729] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e1ac7a94414f268bc12640cdccba09, State: Initialized, Role: FOLLOWER
I20260926 12:28:53.466974 14950 consensus_queue.cc:260] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.467931 14950 ts_tablet_manager.cc:1428] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260926 12:28:53.468243 14917 heartbeater.cc:503] Master 127.7.99.62:43561 was elected leader, sending a full tablet report...
W20260926 12:28:53.513257 14646 tablet.cc:2369] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:53.593922 14957 raft_consensus.cc:482] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:53.594131 14957 raft_consensus.cc:504] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.595304 14957 leader_election.cc:288] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519), 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:53.599507 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.595673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51483 (local address 127.7.99.3:34519)
0926 12:28:53.595813 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.595818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.595974 (+ 156us) server_negotiation.cc:407] Connection header received
0926 12:28:53.596176 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.596181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.596235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.596295 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.596705 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.597237 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.598497 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.598969 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.599080 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.599143 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.599326 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.599387 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:53.599516 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.595599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:34519 (local address 127.7.99.1:51483)
0926 12:28:53.595872 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.595885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.596027 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.596462 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.596465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.596482 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.596611 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.596618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.597348 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.597354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.598375 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.598384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.599081 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.599084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.599184 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.599209 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.599300 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.599372 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":18,"thread_start_us":94,"threads_started":1}
I20260926 12:28:53.599614 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.596114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:55419 (local address 127.7.99.2:44669)
0926 12:28:53.596251 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.596258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.596301 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:28:53.596442 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.596446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.596500 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.596574 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:53.596798 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.597573 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.598651 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.599062 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.599179 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.599236 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.599413 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.599480 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:28:53.599838 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.596015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44669 (local address 127.7.99.1:55419)
0926 12:28:53.596245 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:53.596255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:53.596360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:53.596589 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:53.596591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:53.596603 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:53.596687 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.596690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.597694 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.597697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.598522 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:53.598531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.599184 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.599188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:53.599290 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.599307 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:28:53.599394 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:28:53.599461 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":15,"thread_start_us":80,"threads_started":1}
I20260926 12:28:53.600346 14870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed52210afb541b384d47a9e1f849440" candidate_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e1ac7a94414f268bc12640cdccba09" is_pre_election: true
I20260926 12:28:53.600525 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed52210afb541b384d47a9e1f849440" candidate_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fdeb060b434521b8aa5571c0e82196" is_pre_election: true
I20260926 12:28:53.600639 14870 raft_consensus.cc:2454] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ed588c4ac14303bd5c9c25d8b5afe5 in term 0.
I20260926 12:28:53.600812 14734 raft_consensus.cc:2454] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ed588c4ac14303bd5c9c25d8b5afe5 in term 0.
I20260926 12:28:53.601133 14530 leader_election.cc:302] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ed588c4ac14303bd5c9c25d8b5afe5, b8e1ac7a94414f268bc12640cdccba09; no voters:
I20260926 12:28:53.601610 14957 raft_consensus.cc:2781] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:53.601676 14957 raft_consensus.cc:482] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:53.601728 14957 raft_consensus.cc:3037] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:53.602686 14957 raft_consensus.cc:504] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.603021 14957 leader_election.cc:288] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [CANDIDATE]: Term 1 election: Requested vote from peers b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519), 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669)
I20260926 12:28:53.603348 14870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed52210afb541b384d47a9e1f849440" candidate_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e1ac7a94414f268bc12640cdccba09"
I20260926 12:28:53.603426 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed52210afb541b384d47a9e1f849440" candidate_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fdeb060b434521b8aa5571c0e82196"
I20260926 12:28:53.603515 14870 raft_consensus.cc:3037] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:53.603576 14734 raft_consensus.cc:3037] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:53.604617 14870 raft_consensus.cc:2454] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ed588c4ac14303bd5c9c25d8b5afe5 in term 1.
I20260926 12:28:53.604725 14734 raft_consensus.cc:2454] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ed588c4ac14303bd5c9c25d8b5afe5 in term 1.
I20260926 12:28:53.604907 14530 leader_election.cc:302] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ed588c4ac14303bd5c9c25d8b5afe5, b8e1ac7a94414f268bc12640cdccba09; no voters:
I20260926 12:28:53.605090 14957 raft_consensus.cc:2781] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:53.605249 14957 raft_consensus.cc:686] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 LEADER]: Becoming Leader. State: Replica: 79ed588c4ac14303bd5c9c25d8b5afe5, State: Running, Role: LEADER
I20260926 12:28:53.605587 14957 consensus_queue.cc:237] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:53.608530 13680 catalog_manager.cc:5495] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } health_report { overall_health: HEALTHY } } }
I20260926 12:28:53.625641 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.621978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.7.99.1:38113)
0926 12:28:53.622107 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.622112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.622125 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:28:53.622287 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.622294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.622343 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.622440 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:53.622667 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.623252 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.624216 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.624445 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.624559 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.624850 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:53.625244 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:53.625372 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.625433 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.625504 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:53.631445 14870 raft_consensus.cc:1261] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Refusing update from remote peer 79ed588c4ac14303bd5c9c25d8b5afe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:53.631445 14734 raft_consensus.cc:1261] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Refusing update from remote peer 79ed588c4ac14303bd5c9c25d8b5afe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:53.632274 14957 consensus_queue.cc:1035] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:53.632527 14961 consensus_queue.cc:1035] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:28:53.633996 14919 tablet.cc:2369] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:53.641975 14965 mvcc.cc:205] Tried to move back new op lower bound from 7333583804946800640 to 7333583804850982912. Current Snapshot: MvccSnapshot[applied={T|T < 7333583804946800640}]
I20260926 12:28:53.645979 14966 mvcc.cc:205] Tried to move back new op lower bound from 7333583804946800640 to 7333583804850982912. Current Snapshot: MvccSnapshot[applied={T|T < 7333583804946800640}]
I20260926 12:28:53.649688 13680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:53.649844 13680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:53.654458 13680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08ec1828ecc2448ca760196b3cd9599e: Sending DeleteTablet for 3 replicas of tablet aed52210afb541b384d47a9e1f849440
I20260926 12:28:53.655339 14714 tablet_service.cc:1514] Processing DeleteTablet for tablet aed52210afb541b384d47a9e1f849440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:53 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:43450
I20260926 12:28:53.655536 14850 tablet_service.cc:1514] Processing DeleteTablet for tablet aed52210afb541b384d47a9e1f849440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:53 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:46516
I20260926 12:28:53.656101 14577 tablet_service.cc:1514] Processing DeleteTablet for tablet aed52210afb541b384d47a9e1f849440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:53 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:52726
I20260926 12:28:53.656145 14973 tablet_replica.cc:330] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: stopping tablet replica
I20260926 12:28:53.656555 14973 raft_consensus.cc:2229] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:53.657071 14976 tablet_replica.cc:330] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: stopping tablet replica
I20260926 12:28:53.657232 14973 raft_consensus.cc:2258] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:53.657342 14976 raft_consensus.cc:2229] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:53.657732 14974 tablet_replica.cc:330] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: stopping tablet replica
I20260926 12:28:53.657765 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 13649
I20260926 12:28:53.657770 14976 raft_consensus.cc:2258] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:53.659286 14974 raft_consensus.cc:2229] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:53.660140 14976 ts_tablet_manager.cc:1905] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:53.661041 14974 raft_consensus.cc:2258] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:53.662671 14974 ts_tablet_manager.cc:1905] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:53.662793 14976 ts_tablet_manager.cc:1918] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:53.662869 14976 log.cc:1196] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/aed52210afb541b384d47a9e1f849440
I20260926 12:28:53.663215 14976 ts_tablet_manager.cc:1939] T aed52210afb541b384d47a9e1f849440 P 79ed588c4ac14303bd5c9c25d8b5afe5: Deleting consensus metadata
I20260926 12:28:53.666144 14974 ts_tablet_manager.cc:1918] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:53.666252 14974 log.cc:1196] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/aed52210afb541b384d47a9e1f849440
W20260926 12:28:53.666474 14531 connection.cc:535] server connection from 127.0.0.1:52726 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:28:53.666620 14974 ts_tablet_manager.cc:1939] T aed52210afb541b384d47a9e1f849440 P b8e1ac7a94414f268bc12640cdccba09: Deleting consensus metadata
W20260926 12:28:53.667894 14806 connection.cc:441] server connection from 127.0.0.1:46516 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46516 (request call id 1) could send its response
I20260926 12:28:53.669538 14973 ts_tablet_manager.cc:1905] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:53.672508 14973 ts_tablet_manager.cc:1918] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:53.672613 14973 log.cc:1196] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/aed52210afb541b384d47a9e1f849440
I20260926 12:28:53.672998 14973 ts_tablet_manager.cc:1939] T aed52210afb541b384d47a9e1f849440 P 32fdeb060b434521b8aa5571c0e82196: Deleting consensus metadata
W20260926 12:28:53.674178 14668 connection.cc:441] server connection from 127.0.0.1:43450 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43450 (request call id 1) could send its response
I20260926 12:28:53.676594 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.656885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.7.99.61:44375)
0926 12:28:53.656993 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.656998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.658991 (+ 1993us) server_negotiation.cc:407] Connection header received
0926 12:28:53.659637 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.659643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.659707 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.659766 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.667803 (+ 8037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.668701 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.669920 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.670228 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.670306 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.672484 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.672488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.672491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.672519 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.674449 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.675036 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.675040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.675042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.675131 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.675460 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.675464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.675466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.675750 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.675888 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.676306 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.676426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 12:28:53.684300 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.672934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.7.99.60:34453)
0926 12:28:53.673053 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.673059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.674645 (+ 1586us) server_negotiation.cc:407] Connection header received
0926 12:28:53.674775 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.674778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.674826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.674901 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.676034 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.676861 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.678325 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.678687 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.678756 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.680606 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.680610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.680613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.680638 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.682512 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.682944 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.682947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.682949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.683034 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.683300 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.683303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.683305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.683485 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.683583 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.683895 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.684029 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:28:53.685992 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.673361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.7.99.61:44375)
0926 12:28:53.673485 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.673488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.674984 (+ 1496us) server_negotiation.cc:407] Connection header received
0926 12:28:53.675145 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.675149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.675199 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.675250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.676272 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.677077 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.678916 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.679294 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.679390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.681326 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.681330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.681333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.681359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.683136 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.683716 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.683723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.683727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.683851 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.684443 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.684449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.684451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.684673 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.684785 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.685683 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.685799 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:28:53.688043 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:53.656724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.7.99.60:34453)
0926 12:28:53.656844 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:53.656850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:53.677452 (+ 20602us) server_negotiation.cc:407] Connection header received
0926 12:28:53.677624 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:53.677627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:53.677698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:53.677754 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:53.678938 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.680404 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.682024 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:53.682490 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:53.682576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:53.684349 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:53.684356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:53.684363 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:53.684397 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:53.686171 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.686720 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.686724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.686726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.686842 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:53.687169 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:53.687173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:53.687175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:53.687347 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:53.687487 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:53.687729 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 12:28:53.687863 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 12:28:54.645015 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:54.644570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:38449)
0926 12:28:54.644800 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:54.644920 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
W20260926 12:28:54.645320 14781 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:54.648681 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:54.648211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:53573)
0926 12:28:54.648490 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:54.648583 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
W20260926 12:28:54.648963 14644 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:54.649328 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:54.648887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:59921)
0926 12:28:54.649157 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:54.649232 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
W20260926 12:28:54.649612 14917 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.155040 14987 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08ec1828ecc2448ca760196b3cd9599e)
I20260926 12:28:55.155282 14987 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:55.155789 14988 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08ec1828ecc2448ca760196b3cd9599e)
I20260926 12:28:55.155962 14988 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:55.156569 14987 leader_election.cc:288] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61:44375)
I20260926 12:28:55.156996 14988 leader_election.cc:288] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), b9d1d108102c402d8749c233786a396e (127.7.99.60:34453)
I20260926 12:28:55.157177 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.156720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57194)
0926 12:28:55.156984 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.157061 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260926 12:28:55.157352 14449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.157860 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.157343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57202)
0926 12:28:55.157682 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.157772 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":60,"threads_started":1}
W20260926 12:28:55.158025 14267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.158835 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.158024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57216)
0926 12:28:55.158722 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.158771 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644}
W20260926 12:28:55.159013 14449 leader_election.cc:334] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.159044 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.158688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57218)
0926 12:28:55.158914 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.158972 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260926 12:28:55.159126 14267 leader_election.cc:334] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.161522 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.157049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.7.99.61:44375)
0926 12:28:55.157441 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:55.157457 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:55.157871 (+ 414us) server_negotiation.cc:407] Connection header received
0926 12:28:55.158105 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:55.158110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:55.158163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:55.158222 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:55.158573 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.159434 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.160539 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.160941 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.161055 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:55.161111 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:55.161278 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 12:28:55.161379 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":327,"thread_start_us":63,"threads_started":1}
I20260926 12:28:55.161516 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.156773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:44375 (local address 127.0.0.1:60068)
0926 12:28:55.157786 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.157800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:55.157984 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:55.158367 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:55.158370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:55.158392 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:55.158475 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:55.158481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.159555 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.159560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:55.160417 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:55.160425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.161055 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.161058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:55.161149 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:55.161170 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:28:55.161265 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:28:55.161343 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"spinlock_wait_cycles":768,"thread_start_us":79,"threads_started":1}
I20260926 12:28:55.161522 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.157302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.7.99.60:34453)
0926 12:28:55.158153 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:55.158157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:55.158169 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:55.158212 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:55.158216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:55.158251 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:55.158333 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:55.158624 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.159426 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.160534 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.160940 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.161053 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:55.161111 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:55.161274 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:28:55.161361 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":748,"thread_start_us":58,"threads_started":1}
I20260926 12:28:55.161537 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.157159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:34453 (local address 127.0.0.1:56256)
0926 12:28:55.157620 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.157646 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:55.157774 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:55.158437 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:55.158440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:55.158455 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:55.158541 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:55.158550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.159535 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.159539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:55.160415 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:55.160423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:55.161053 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:55.161057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:55.161145 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:55.161165 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:28:55.161260 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:28:55.161369 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":30,"thread_start_us":63,"threads_started":1}
I20260926 12:28:55.162108 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d1d108102c402d8749c233786a396e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" is_pre_election: true
I20260926 12:28:55.162340 14284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d1d108102c402d8749c233786a396e in term 1.
I20260926 12:28:55.162472 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9d1d108102c402d8749c233786a396e" is_pre_election: true
I20260926 12:28:55.162671 14474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d18ca25f6ac448fa8e21839e6a4ba216 in term 1.
I20260926 12:28:55.162714 14451 leader_election.cc:302] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d1d108102c402d8749c233786a396e, d18ca25f6ac448fa8e21839e6a4ba216; no voters: 08ec1828ecc2448ca760196b3cd9599e
I20260926 12:28:55.162948 14987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:28:55.162997 14987 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08ec1828ecc2448ca760196b3cd9599e)
I20260926 12:28:55.163034 14987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:28:55.163028 14268 leader_election.cc:302] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d1d108102c402d8749c233786a396e, d18ca25f6ac448fa8e21839e6a4ba216; no voters: 08ec1828ecc2448ca760196b3cd9599e
I20260926 12:28:55.163206 14988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:28:55.163251 14988 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08ec1828ecc2448ca760196b3cd9599e)
I20260926 12:28:55.163290 14988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:28:55.164016 14987 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:55.164340 14988 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:55.164409 14987 leader_election.cc:288] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 election: Requested vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61:44375)
I20260926 12:28:55.164666 14988 leader_election.cc:288] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 election: Requested vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), b9d1d108102c402d8749c233786a396e (127.7.99.60:34453)
I20260926 12:28:55.164856 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.164506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57222)
0926 12:28:55.164747 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.164794 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260926 12:28:55.165001 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9d1d108102c402d8749c233786a396e"
I20260926 12:28:55.165015 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d1d108102c402d8749c233786a396e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
I20260926 12:28:55.165149 14474 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d18ca25f6ac448fa8e21839e6a4ba216 in current term 2: Already voted for candidate b9d1d108102c402d8749c233786a396e in this term.
I20260926 12:28:55.165166 14284 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d1d108102c402d8749c233786a396e in current term 2: Already voted for candidate d18ca25f6ac448fa8e21839e6a4ba216 in this term.
I20260926 12:28:55.165606 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.164758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57234)
0926 12:28:55.165496 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.165546 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679}
I20260926 12:28:55.165622 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.165255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57244)
0926 12:28:55.165530 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.165571 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 12:28:55.165740 14449 leader_election.cc:334] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.165810 14449 leader_election.cc:302] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d1d108102c402d8749c233786a396e; no voters: 08ec1828ecc2448ca760196b3cd9599e, d18ca25f6ac448fa8e21839e6a4ba216
I20260926 12:28:55.165930 14987 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:28:55.166184 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.165965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57248)
0926 12:28:55.166102 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.166143 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 12:28:55.166302 14267 leader_election.cc:334] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:55.166374 14267 leader_election.cc:302] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ca25f6ac448fa8e21839e6a4ba216; no voters: 08ec1828ecc2448ca760196b3cd9599e, b9d1d108102c402d8749c233786a396e
I20260926 12:28:55.166538 14988 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:28:55.646302 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.645876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:53365)
0926 12:28:55.646157 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.646212 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260926 12:28:55.649816 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.649391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:33419)
0926 12:28:55.649660 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.649723 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260926 12:28:55.650379 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:55.650075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:38083)
0926 12:28:55.650264 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:55.650311 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260926 12:28:56.651278 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:56.650836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:37959)
0926 12:28:56.651089 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:56.651161 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260926 12:28:56.652511 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:56.649618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:41613)
0926 12:28:56.652311 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:56.652395 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2607,"thread_start_us":92,"threads_started":1}
I20260926 12:28:56.655421 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:56.655010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:46061)
0926 12:28:56.655257 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:56.655324 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260926 12:28:57.195207 15004 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:57.195391 15004 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:57.195906 15004 leader_election.cc:288] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61:44375)
I20260926 12:28:57.196391 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d1d108102c402d8749c233786a396e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" is_pre_election: true
I20260926 12:28:57.196512 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.196060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57346)
0926 12:28:57.196350 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.196418 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":71,"threads_started":1}
I20260926 12:28:57.196545 14284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d1d108102c402d8749c233786a396e in term 2.
I20260926 12:28:57.196875 14451 leader_election.cc:302] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d1d108102c402d8749c233786a396e, d18ca25f6ac448fa8e21839e6a4ba216; no voters:
I20260926 12:28:57.197032 15004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 12:28:57.197094 15004 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:57.197153 15004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:28:57.197412 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.197230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57350)
0926 12:28:57.197311 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.197357 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 12:28:57.197554 14449 leader_election.cc:334] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:57.198319 15004 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:57.198633 15004 leader_election.cc:288] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 election: Requested vote from peers 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561), d18ca25f6ac448fa8e21839e6a4ba216 (127.7.99.61:44375)
I20260926 12:28:57.198966 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d1d108102c402d8749c233786a396e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d18ca25f6ac448fa8e21839e6a4ba216"
I20260926 12:28:57.199057 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.198722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57366)
0926 12:28:57.198950 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.198999 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 12:28:57.199088 14284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:28:57.199978 14284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d1d108102c402d8749c233786a396e in term 3.
I20260926 12:28:57.200268 14451 leader_election.cc:302] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d1d108102c402d8749c233786a396e, d18ca25f6ac448fa8e21839e6a4ba216; no voters:
I20260926 12:28:57.200456 15004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:28:57.200641 15004 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [term 3 LEADER]: Becoming Leader. State: Replica: b9d1d108102c402d8749c233786a396e, State: Running, Role: LEADER
I20260926 12:28:57.200727 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.200500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57378)
0926 12:28:57.200605 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.200650 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:28:57.200858 14449 leader_election.cc:334] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
I20260926 12:28:57.200958 15004 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } }
I20260926 12:28:57.202105 15007 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d1d108102c402d8749c233786a396e. Latest consensus state: current_term: 3 leader_uuid: "b9d1d108102c402d8749c233786a396e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:57.202193 15007 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:57.202421 15009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:57.204641 15009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:57.205037 15009 catalog_manager.cc:1259] Loaded cluster ID: 87bcdefce27148d6aacf059caa76cbbf
I20260926 12:28:57.205104 15009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:57.205643 15009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:57.206020 15009 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Loaded TSK: 0
I20260926 12:28:57.206697 15009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:28:57.221472 14464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:57.224792 14464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:57.227859 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.227600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57406)
0926 12:28:57.227725 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.227778 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:28:57.227931 14284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [term 3 FOLLOWER]: Refusing update from remote peer b9d1d108102c402d8749c233786a396e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 12:28:57.228426 15007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 12:28:57.228602 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.228392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.0.0.1:57414)
0926 12:28:57.228482 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.228536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 12:28:57.228758 14449 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e -> Peer 08ec1828ecc2448ca760196b3cd9599e (127.7.99.62:43561): Couldn't send request to peer 08ec1828ecc2448ca760196b3cd9599e. Status: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 12:28:57.230010 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d1d108102c402d8749c233786a396e. Latest consensus state: current_term: 3 leader_uuid: "b9d1d108102c402d8749c233786a396e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:57.230140 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:57.231279 15004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b9d1d108102c402d8749c233786a396e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:57.231387 15004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:57.231570 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b9d1d108102c402d8749c233786a396e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "08ec1828ecc2448ca760196b3cd9599e" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43561 } } peers { permanent_uuid: "d18ca25f6ac448fa8e21839e6a4ba216" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 44375 } attrs { promote: false } } peers { permanent_uuid: "b9d1d108102c402d8749c233786a396e" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 34453 } attrs { promote: false } } }
I20260926 12:28:57.231695 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ca25f6ac448fa8e21839e6a4ba216 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:28:57.232236 15012 mvcc.cc:205] Tried to move back new op lower bound from 7333583819680264192 to 7333583819577409536. Current Snapshot: MvccSnapshot[applied={T|T < 7333583819680264192}]
I20260926 12:28:57.242228 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.237220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:34519 (local address 127.0.0.1:53008)
0926 12:28:57.237307 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.237321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:57.237412 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:57.238503 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:57.238506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:57.238522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:57.238606 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.238612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239743 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.239746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.240986 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.240994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.241825 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.241829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.241891 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.241910 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:28:57.242020 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:28:57.242083 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":78}
I20260926 12:28:57.242245 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.237994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.7.99.3:34519)
0926 12:28:57.238245 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.238249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.238264 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:28:57.238311 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.238314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.238366 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.238708 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:57.238810 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239628 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.241104 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.241710 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.241817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.241880 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.242044 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.242122 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260926 12:28:57.242540 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.237573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.7.99.2:44669)
0926 12:28:57.237857 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.237863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.237877 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:28:57.237930 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.237932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.237979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.238059 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:57.238384 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239116 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.241270 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.241885 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.242025 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.242084 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.242289 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.242378 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":65,"threads_started":1}
I20260926 12:28:57.242913 14850 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d00dbdd61489b8003607f2612953c (DEFAULT_TABLE table=test-table [id=0ad8ccaba6f340b1a40e13d87cfd48b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:57.243055 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.237451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44669 (local address 127.0.0.1:54354)
0926 12:28:57.237690 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.237703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:57.237777 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:57.238095 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:57.238098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:57.238116 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:57.238230 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.238238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239289 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.239294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.241154 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.241162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.242013 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.242018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.242090 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.242108 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:28:57.242170 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 12:28:57.242222 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":76,"thread_start_us":90,"threads_started":1}
I20260926 12:28:57.243192 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d00dbdd61489b8003607f2612953c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:57.243220 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.237755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38113 (local address 127.0.0.1:52676)
0926 12:28:57.238336 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.238349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:57.238433 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:57.238748 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:57.238751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:57.238767 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:57.238844 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.238849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239819 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.239822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.241152 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.241162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.242877 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.242880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.242940 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.242956 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:28:57.243039 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:28:57.243105 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260926 12:28:57.243578 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.238285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.7.99.1:38113)
0926 12:28:57.238530 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.238534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.238546 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:57.238591 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.238593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.238641 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.238723 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:57.238958 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.239671 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.242137 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.242664 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.242772 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.242832 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.243056 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.243455 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":57,"threads_started":1}
I20260926 12:28:57.244025 14577 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d00dbdd61489b8003607f2612953c (DEFAULT_TABLE table=test-table [id=0ad8ccaba6f340b1a40e13d87cfd48b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:57.244275 14577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d00dbdd61489b8003607f2612953c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:57.244436 14714 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d00dbdd61489b8003607f2612953c (DEFAULT_TABLE table=test-table [id=0ad8ccaba6f340b1a40e13d87cfd48b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:57.244664 14714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d00dbdd61489b8003607f2612953c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:57.245671 15022 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Bootstrap starting.
I20260926 12:28:57.246282 15023 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: Bootstrap starting.
I20260926 12:28:57.246549 15022 tablet_bootstrap.cc:654] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:57.246990 15024 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Bootstrap starting.
I20260926 12:28:57.247225 15023 tablet_bootstrap.cc:654] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:57.247797 15022 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: No bootstrap required, opened a new log
I20260926 12:28:57.247881 15022 ts_tablet_manager.cc:1397] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:57.248365 15022 raft_consensus.cc:348] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.248481 15022 raft_consensus.cc:374] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:57.248531 15022 raft_consensus.cc:729] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e1ac7a94414f268bc12640cdccba09, State: Initialized, Role: FOLLOWER
I20260926 12:28:57.248579 15024 tablet_bootstrap.cc:654] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:57.248636 15023 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: No bootstrap required, opened a new log
I20260926 12:28:57.248653 15022 consensus_queue.cc:260] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.248746 15023 ts_tablet_manager.cc:1397] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:57.248996 15022 ts_tablet_manager.cc:1428] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:57.249164 15023 raft_consensus.cc:348] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.249316 15023 raft_consensus.cc:374] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:57.249406 15023 raft_consensus.cc:729] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ed588c4ac14303bd5c9c25d8b5afe5, State: Initialized, Role: FOLLOWER
I20260926 12:28:57.249598 15023 consensus_queue.cc:260] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.249961 15023 ts_tablet_manager.cc:1428] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:57.250161 15024 tablet_bootstrap.cc:492] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: No bootstrap required, opened a new log
I20260926 12:28:57.250288 15024 ts_tablet_manager.cc:1397] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:28:57.250663 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.249620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:38065)
0926 12:28:57.250484 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.250552 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":81,"threads_started":1}
I20260926 12:28:57.251266 14916 heartbeater.cc:503] Master 127.7.99.60:34453 was elected leader, sending a full tablet report...
I20260926 12:28:57.251459 15024 raft_consensus.cc:348] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.251545 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.251186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:52193)
0926 12:28:57.251399 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.251453 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260926 12:28:57.251886 14643 heartbeater.cc:503] Master 127.7.99.60:34453 was elected leader, sending a full tablet report...
I20260926 12:28:57.252478 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.252243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:50733)
0926 12:28:57.252372 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.252421 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 12:28:57.252810 15024 raft_consensus.cc:374] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:57.252875 15024 raft_consensus.cc:729] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fdeb060b434521b8aa5571c0e82196, State: Initialized, Role: FOLLOWER
I20260926 12:28:57.253131 15024 consensus_queue.cc:260] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.253536 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.253235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:49595)
0926 12:28:57.253429 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.253476 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 12:28:57.253537 15024 ts_tablet_manager.cc:1428] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:57.254345 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.253991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:47713)
0926 12:28:57.254210 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.254264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260926 12:28:57.254895 14780 heartbeater.cc:503] Master 127.7.99.60:34453 was elected leader, sending a full tablet report...
I20260926 12:28:57.255277 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.255017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:38665)
0926 12:28:57.255151 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.255197 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 12:28:57.432349 15027 raft_consensus.cc:482] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:57.432530 15027 raft_consensus.cc:504] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.433650 15027 leader_election.cc:288] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669), 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
I20260926 12:28:57.437354 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.434083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35605 (local address 127.7.99.2:44669)
0926 12:28:57.434206 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.434209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.434242 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:28:57.434377 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.434387 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.434431 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.434533 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:57.434795 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.435300 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.436398 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.436791 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.436919 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.436993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.437167 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.437236 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:28:57.437369 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.433977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44669 (local address 127.7.99.3:35605)
0926 12:28:57.434123 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.434140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:57.434252 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:57.434549 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:57.434552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:57.434573 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:57.434677 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.434690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.435413 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.435418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.436278 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.436287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.436932 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.436936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.437039 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.437061 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:28:57.437154 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 12:28:57.437221 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260926 12:28:57.437909 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d00dbdd61489b8003607f2612953c" candidate_uuid: "b8e1ac7a94414f268bc12640cdccba09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fdeb060b434521b8aa5571c0e82196" is_pre_election: true
I20260926 12:28:57.438081 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.434478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35485 (local address 127.7.99.1:38113)
0926 12:28:57.434668 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.434673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.434791 (+ 118us) server_negotiation.cc:407] Connection header received
0926 12:28:57.434997 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.435001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.435063 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.435157 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:57.435436 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.435934 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.437032 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.437509 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.437637 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.437700 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.437895 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.437963 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 12:28:57.438082 14734 raft_consensus.cc:2454] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e1ac7a94414f268bc12640cdccba09 in term 0.
I20260926 12:28:57.438129 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.434294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38113 (local address 127.7.99.3:35485)
0926 12:28:57.434674 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.434690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:57.434878 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:57.435168 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:57.435172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:57.435188 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:57.435315 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.435322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.436056 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.436063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.436897 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:57.436906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.437637 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.437641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:57.437746 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.437773 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:28:57.437882 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:28:57.437958 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":152,"threads_started":1}
I20260926 12:28:57.438489 14803 leader_election.cc:302] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fdeb060b434521b8aa5571c0e82196, b8e1ac7a94414f268bc12640cdccba09; no voters:
I20260926 12:28:57.438652 14597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d00dbdd61489b8003607f2612953c" candidate_uuid: "b8e1ac7a94414f268bc12640cdccba09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" is_pre_election: true
I20260926 12:28:57.438760 15027 raft_consensus.cc:2781] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:57.438812 15027 raft_consensus.cc:482] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:57.438845 15027 raft_consensus.cc:3037] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:57.438894 14597 raft_consensus.cc:2454] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e1ac7a94414f268bc12640cdccba09 in term 0.
I20260926 12:28:57.439882 15027 raft_consensus.cc:504] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.440306 15027 leader_election.cc:288] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [CANDIDATE]: Term 1 election: Requested vote from peers 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669), 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113)
I20260926 12:28:57.440672 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d00dbdd61489b8003607f2612953c" candidate_uuid: "b8e1ac7a94414f268bc12640cdccba09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fdeb060b434521b8aa5571c0e82196"
I20260926 12:28:57.440788 14734 raft_consensus.cc:3037] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:57.440766 14597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d00dbdd61489b8003607f2612953c" candidate_uuid: "b8e1ac7a94414f268bc12640cdccba09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5"
I20260926 12:28:57.440889 14597 raft_consensus.cc:3037] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:57.441670 14734 raft_consensus.cc:2454] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e1ac7a94414f268bc12640cdccba09 in term 1.
I20260926 12:28:57.441910 14597 raft_consensus.cc:2454] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e1ac7a94414f268bc12640cdccba09 in term 1.
I20260926 12:28:57.441931 14803 leader_election.cc:302] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fdeb060b434521b8aa5571c0e82196, b8e1ac7a94414f268bc12640cdccba09; no voters:
I20260926 12:28:57.442186 15027 raft_consensus.cc:2781] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:57.442374 15027 raft_consensus.cc:686] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 1 LEADER]: Becoming Leader. State: Replica: b8e1ac7a94414f268bc12640cdccba09, State: Running, Role: LEADER
I20260926 12:28:57.442646 15027 consensus_queue.cc:237] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } }
I20260926 12:28:57.443209 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.442982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:37833)
0926 12:28:57.443096 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.443148 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:28:57.445516 14463 catalog_manager.cc:5495] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "b8e1ac7a94414f268bc12640cdccba09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e1ac7a94414f268bc12640cdccba09" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 34519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 } health_report { overall_health: UNKNOWN } } }
W20260926 12:28:57.448335 14783 tablet.cc:2369] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:57.475934 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.472425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.7.99.3:34519)
0926 12:28:57.472553 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:57.472558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:57.472571 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:28:57.472722 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:57.472725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:57.472779 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:57.472897 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:57.473189 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.473756 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.474583 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:57.474858 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:57.474948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:57.475162 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:57.475609 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:57.475692 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:57.475751 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 12:28:57.475830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:28:57.478608 14734 raft_consensus.cc:1261] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Refusing update from remote peer b8e1ac7a94414f268bc12640cdccba09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:57.478855 14597 raft_consensus.cc:1261] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5 [term 1 FOLLOWER]: Refusing update from remote peer b8e1ac7a94414f268bc12640cdccba09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:57.479113 15035 consensus_queue.cc:1035] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fdeb060b434521b8aa5571c0e82196" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:57.479350 15027 consensus_queue.cc:1035] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ed588c4ac14303bd5c9c25d8b5afe5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:57.481359 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.481048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:39183)
0926 12:28:57.481197 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.481253 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 12:28:57.481565 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.481269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:58881)
0926 12:28:57.481430 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.481487 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 12:28:57.483830 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.483574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:60117)
0926 12:28:57.483690 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.483739 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 12:28:57.483798 15041 mvcc.cc:205] Tried to move back new op lower bound from 7333583820707405824 to 7333583820567883776. Current Snapshot: MvccSnapshot[applied={T|T < 7333583820707405824}]
I20260926 12:28:57.484515 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.484292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.1:41927)
0926 12:28:57.484418 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.484462 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 12:28:57.486390 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.486164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.3:40503)
0926 12:28:57.486260 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.486316 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 12:28:57.487738 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:57.487479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43561 (local address 127.7.99.2:55275)
0926 12:28:57.487637 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.487681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 12:28:57.492091 14463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:57.492395 14463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:57.496047 14463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9d1d108102c402d8749c233786a396e: Sending DeleteTablet for 3 replicas of tablet fe3d00dbdd61489b8003607f2612953c
I20260926 12:28:57.496748 14714 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d00dbdd61489b8003607f2612953c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:57 UTC) from {username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:54354
I20260926 12:28:57.496867 14850 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d00dbdd61489b8003607f2612953c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:57 UTC) from {username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:53008
I20260926 12:28:57.497037 14577 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d00dbdd61489b8003607f2612953c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:57 UTC) from {username='oryx', principal='oryx/127.7.99.60@KRBTEST.COM'} at 127.0.0.1:52676
I20260926 12:28:57.497083 15048 tablet_replica.cc:330] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: stopping tablet replica
I20260926 12:28:57.497200 15048 raft_consensus.cc:2229] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:57.497330 15048 raft_consensus.cc:2258] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:57.497360 15049 tablet_replica.cc:330] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: stopping tablet replica
I20260926 12:28:57.497484 15049 raft_consensus.cc:2229] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:57.497751 15049 raft_consensus.cc:2258] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:57.497977 15050 tablet_replica.cc:330] T fe3d00dbdd61489b8003607f2612953c P 79ed588c4ac14303bd5c9c25d8b5afe5: stopping tablet replica
I20260926 12:28:57.498027 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14426
I20260926 12:28:57.499045 15049 ts_tablet_manager.cc:1905] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:57.502316 15048 ts_tablet_manager.cc:1905] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:57.502985 15049 ts_tablet_manager.cc:1918] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:57.503127 15049 log.cc:1196] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/fe3d00dbdd61489b8003607f2612953c
I20260926 12:28:57.503489 15049 ts_tablet_manager.cc:1939] T fe3d00dbdd61489b8003607f2612953c P b8e1ac7a94414f268bc12640cdccba09: Deleting consensus metadata
I20260926 12:28:57.504236 14448 catalog_manager.cc:4841] TS b8e1ac7a94414f268bc12640cdccba09 (127.7.99.3:34519): tablet fe3d00dbdd61489b8003607f2612953c (table test-table [id=0ad8ccaba6f340b1a40e13d87cfd48b6]) successfully deleted
I20260926 12:28:57.505621 15048 ts_tablet_manager.cc:1918] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:57.505704 15048 log.cc:1196] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/fe3d00dbdd61489b8003607f2612953c
I20260926 12:28:57.505993 15048 ts_tablet_manager.cc:1939] T fe3d00dbdd61489b8003607f2612953c P 32fdeb060b434521b8aa5571c0e82196: Deleting consensus metadata
I20260926 12:28:57.506836 14448 catalog_manager.cc:4841] TS 32fdeb060b434521b8aa5571c0e82196 (127.7.99.2:44669): tablet fe3d00dbdd61489b8003607f2612953c (table test-table [id=0ad8ccaba6f340b1a40e13d87cfd48b6]) successfully deleted
W20260926 12:28:57.506958 15017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:57.506644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38113 (local address 127.0.0.1:52682)
0926 12:28:57.506774 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:57.506843 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38113: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 12:28:57.507097 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14651
W20260926 12:28:57.507225 14448 catalog_manager.cc:4584] TS 79ed588c4ac14303bd5c9c25d8b5afe5 (127.7.99.1:38113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe3d00dbdd61489b8003607f2612953c: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38113: connect: Connection reset by peer (error 104)
I20260926 12:28:57.512337 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14787
I20260926 12:28:57.518290 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14145
I20260926 12:28:57.524137 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 14254
2026-09-26T12:28:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7431 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:28:57 dist-test-slave-3jw2 krb5kdc[15053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:28:57 dist-test-slave-3jw2 krb5kdc[15053](info): set up 2 sockets
Sep 26 12:28:57 dist-test-slave-3jw2 krb5kdc[15053](info): commencing operation
krb5kdc: starting...
W20260926 12:28:58.564927 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:28:58 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:28:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab.
Entry for principal oryx/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab.
I20260926 12:28:58.682569 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43313
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab
--principal=oryx/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:58.791060 15066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:58.791296 15066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:58.791334 15066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:58.795189 15066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:28:58.795233 15066 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:28:58.795250 15066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:58.795266 15066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:58.795281 15066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:28:58.795295 15066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:28:58.800482 15066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.62.keytab
--principal=oryx/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:58.801456 15066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:58.802421 15066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:58.811008 15072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:58.811048 15071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:58.811048 15074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:58.811288 15066 server_base.cc:1034] running on GCE node
I20260926 12:28:58.811838 15066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:58.813210 15066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:58.814378 15066 hybrid_clock.cc:648] HybridClock initialized: now 1790425738814350 us; error 47 us; skew 500 ppm
Sep 26 12:28:58 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425738, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:58.817001 15066 init.cc:377] Logged in from keytab as oryx/127.7.99.62@KRBTEST.COM (short username oryx)
I20260926 12:28:58.817849 15066 webserver.cc:460] Webserver started at http://127.7.99.62:36533/ using document root <none> and password file <none>
I20260926 12:28:58.818403 15066 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:58.818490 15066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:58.818719 15066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:58.820704 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "4721ca5fb37b4ef5ae70d7f490c6f680"
format_stamp: "Formatted at 2026-09-26 12:28:58 on dist-test-slave-3jw2"
server_key: "a02977959119a1354af8dede5c1c9558"
server_key_iv: "bf7eed7e99d13acdfeb0a7033867010d"
server_key_version: "encryptionkey@0"
I20260926 12:28:58.821225 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "4721ca5fb37b4ef5ae70d7f490c6f680"
format_stamp: "Formatted at 2026-09-26 12:28:58 on dist-test-slave-3jw2"
server_key: "a02977959119a1354af8dede5c1c9558"
server_key_iv: "bf7eed7e99d13acdfeb0a7033867010d"
server_key_version: "encryptionkey@0"
I20260926 12:28:58.824616 15066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:28:58.826872 15081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:58.828014 15066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:58.828150 15066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "4721ca5fb37b4ef5ae70d7f490c6f680"
format_stamp: "Formatted at 2026-09-26 12:28:58 on dist-test-slave-3jw2"
server_key: "a02977959119a1354af8dede5c1c9558"
server_key_iv: "bf7eed7e99d13acdfeb0a7033867010d"
server_key_version: "encryptionkey@0"
I20260926 12:28:58.828293 15066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:58.846671 15066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:58.849355 15066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:58.849555 15066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:58.857205 15066 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43313
I20260926 12:28:58.857213 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43313 every 8 connection(s)
I20260926 12:28:58.858104 15066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:28:58.860770 15144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:58.865939 15144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Bootstrap starting.
I20260926 12:28:58.868083 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15066
I20260926 12:28:58.868224 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:28:58.868530 7564 external_mini_cluster.cc:1428] Setting key 8a035dbfbb338b1f60d2f4f47636bf72
I20260926 12:28:58.868606 15144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:58.869477 15144 log.cc:824] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:58.871344 15144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: No bootstrap required, opened a new log
I20260926 12:28:58.874624 15144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } }
I20260926 12:28:58.874856 15144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:58.874935 15144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4721ca5fb37b4ef5ae70d7f490c6f680, State: Initialized, Role: FOLLOWER
I20260926 12:28:58.875430 15144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } }
I20260926 12:28:58.875597 15144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:28:58.875664 15144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:28:58.875757 15144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:58.876837 15144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } }
Sep 26 12:28:58 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:58.877199 15144 leader_election.cc:302] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4721ca5fb37b4ef5ae70d7f490c6f680; no voters:
I20260926 12:28:58.877519 15144 leader_election.cc:288] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:28:58.877601 15149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:58.877826 15149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [term 1 LEADER]: Becoming Leader. State: Replica: 4721ca5fb37b4ef5ae70d7f490c6f680, State: Running, Role: LEADER
I20260926 12:28:58.878147 15149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } }
I20260926 12:28:58.878803 15144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:28:58.880571 15150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } } }
I20260926 12:28:58.880707 15150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:58.881013 15151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4721ca5fb37b4ef5ae70d7f490c6f680. Latest consensus state: current_term: 1 leader_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4721ca5fb37b4ef5ae70d7f490c6f680" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43313 } } }
I20260926 12:28:58.881415 15151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680 [sys.catalog]: This master's current role is: LEADER
I20260926 12:28:58.883287 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:58.869713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.7.99.62:43313)
0926 12:28:58.870109 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:58.870123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:58.870155 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:28:58.870765 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:58.870781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:58.871093 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:58.871422 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:58.872210 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:58.873176 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:58.874035 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:58.874501 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:58.874663 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:58.877761 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:58.877770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:58.877792 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:58.877827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:58.879889 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:58.880458 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:58.880467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:58.880475 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:58.880639 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:58.881011 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:58.881014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:58.881015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:58.881535 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:58.881736 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:58.882050 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 12:28:58.882381 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
W20260926 12:28:58.883716 15163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:28:58.883795 15163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:28:58.883879 15164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:28:58.884680 15164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:28:58.889678 15164 catalog_manager.cc:1347] Generated new cluster ID: 3aec429c73bb4626988624c3ef601cd1
I20260926 12:28:58.889736 15164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:28:58.901113 15164 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:28:58.902251 15164 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:28:58.917666 15164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Generated new TSK 0
I20260926 12:28:58.918398 15164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
I20260926 12:28:58.951021 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43313
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:59.058061 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:59.058283 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:59.058324 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:59.062178 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:59.062220 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:59.062256 15170 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:59.062287 15170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:28:59.067222 15170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:59.068116 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:59.069096 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:59.078509 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:59.078528 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:59.078503 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:59.078733 15170 server_base.cc:1034] running on GCE node
I20260926 12:28:59.079260 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:59.079893 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:59.081050 15170 hybrid_clock.cc:648] HybridClock initialized: now 1790425739080985 us; error 76 us; skew 500 ppm
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:59.083758 15170 init.cc:377] Logged in from keytab as oryx/127.7.99.1@KRBTEST.COM (short username oryx)
I20260926 12:28:59.084621 15170 webserver.cc:460] Webserver started at http://127.7.99.1:34809/ using document root <none> and password file <none>
I20260926 12:28:59.085104 15170 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:59.085160 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:59.085311 15170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:59.087193 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "b52234cafd7182c9f70c3720bf9cc327"
server_key_iv: "86e2970e3a1da38c69c2db279f46859b"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.087677 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "b52234cafd7182c9f70c3720bf9cc327"
server_key_iv: "86e2970e3a1da38c69c2db279f46859b"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.091041 15170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:28:59.093199 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.094278 15170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:28:59.094410 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "b52234cafd7182c9f70c3720bf9cc327"
server_key_iv: "86e2970e3a1da38c69c2db279f46859b"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.094519 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:59.111056 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:59.114359 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:59.114573 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:59.115990 15170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:59.116060 15170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.116122 15170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:59.116179 15170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:59.127092 15170 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:45741
I20260926 12:28:59.128067 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:28:59.128341 15300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:45741 every 8 connection(s)
I20260926 12:28:59.130160 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.116653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:46367 (local address 127.7.99.62:43313)
0926 12:28:59.116816 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.116821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.117668 (+ 847us) server_negotiation.cc:407] Connection header received
0926 12:28:59.118583 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.118586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.118642 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.118743 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.120525 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.121127 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.122234 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.122491 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.122605 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.126239 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.126243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.126245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.126268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.127894 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.128559 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.128562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.128564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.128654 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.129044 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.129047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.129049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.129266 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.129375 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.129843 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.129992 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:28:59.130868 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.117070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.1:46367)
0926 12:28:59.117500 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.117548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.118374 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.119074 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.119096 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.119599 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.120277 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.120301 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.121257 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.121261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.122110 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.122119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.122656 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.122660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.122845 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.123514 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.123529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.126064 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.128028 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.128036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.128051 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.128457 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.128766 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.128770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.128772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.128930 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.129380 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.129388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.129618 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.129787 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.130085 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":133,"threads_started":1}
I20260926 12:28:59.132120 15301 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43313
I20260926 12:28:59.132270 15301 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:59.132802 15301 heartbeater.cc:511] Master 127.7.99.62:43313 requested a full tablet report, sending...
I20260926 12:28:59.134387 15097 ts_manager.cc:194] Registered new tserver with Master: 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741)
I20260926 12:28:59.135486 15097 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:46367
I20260926 12:28:59.136770 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15170
I20260926 12:28:59.136996 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:28:59.137290 7564 external_mini_cluster.cc:1428] Setting key 9f081ee0d75ba8e3dd261d0a95b6e90d
I20260926 12:28:59.142867 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.134493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:35415 (local address 127.7.99.62:43313)
0926 12:28:59.134592 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.134598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.136180 (+ 1582us) server_negotiation.cc:407] Connection header received
0926 12:28:59.136346 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.136348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.136383 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.136465 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.137000 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.137716 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.138546 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.138825 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.138937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.139906 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.139911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.139916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.139942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.141270 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.141746 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.141748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.141749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.141811 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.142091 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.142093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.142094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.142231 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.142297 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.142607 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.142692 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:28:59.142977 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.134373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.1:35415)
0926 12:28:59.136078 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.136090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.136240 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.136497 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.136500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.136694 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.136864 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.136872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.137888 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.137891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.138446 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.138453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.138933 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.138935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.139002 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.139472 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.139482 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.139768 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.141384 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.141390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.141394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.141648 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.141896 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.141901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.141906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.142006 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.142319 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.142332 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.142438 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.142561 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.142722 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"mutex_wait_us":47,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
I20260926 12:28:59.163434 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43313
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:59.270780 15310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:59.271024 15310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:59.271068 15310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:59.274997 15310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:59.275054 15310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:59.275091 15310 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:59.275169 15310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:28:59.280557 15310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:59.281543 15310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:59.282425 15310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:59.291988 15315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:59.291992 15318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:59.292171 15310 server_base.cc:1034] running on GCE node
W20260926 12:28:59.291994 15316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:59.292572 15310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:59.293164 15310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:59.294339 15310 hybrid_clock.cc:648] HybridClock initialized: now 1790425739294327 us; error 32 us; skew 500 ppm
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:59.296916 15310 init.cc:377] Logged in from keytab as oryx/127.7.99.2@KRBTEST.COM (short username oryx)
I20260926 12:28:59.297770 15310 webserver.cc:460] Webserver started at http://127.7.99.2:39193/ using document root <none> and password file <none>
I20260926 12:28:59.298374 15310 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:59.298475 15310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:59.298688 15310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:59.300676 15310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "4fe97be3f2564511bb49824c1c1c151c"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "ba5f882e480ece55fb4c5ed7463e28ae"
server_key_iv: "372c53a0a79ece496e3deac6c8457de3"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.301194 15310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "4fe97be3f2564511bb49824c1c1c151c"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "ba5f882e480ece55fb4c5ed7463e28ae"
server_key_iv: "372c53a0a79ece496e3deac6c8457de3"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.304632 15310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:28:59.306881 15325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.308032 15310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:59.308208 15310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "4fe97be3f2564511bb49824c1c1c151c"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "ba5f882e480ece55fb4c5ed7463e28ae"
server_key_iv: "372c53a0a79ece496e3deac6c8457de3"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.308318 15310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:59.321810 15310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:59.324536 15310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:59.324714 15310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:59.326009 15310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:59.326079 15310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.326143 15310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:59.326192 15310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:59.336755 15310 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:40339
I20260926 12:28:59.336786 15440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:40339 every 8 connection(s)
I20260926 12:28:59.337775 15310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:28:59.339708 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15310
I20260926 12:28:59.339844 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:28:59.340202 7564 external_mini_cluster.cc:1428] Setting key 9075a2046224e47fd16674fd6c140284
I20260926 12:28:59.341355 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.327407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.2:49655)
0926 12:28:59.327863 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.327902 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.328903 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.329750 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.329764 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.330229 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.330909 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.330930 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.332143 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.332148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.333797 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.333808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.334287 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.334291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.334501 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.335131 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.335157 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.337020 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.338834 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.338838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.338848 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.339081 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.339305 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.339307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.339308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.339423 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.339835 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.339841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.340105 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.340306 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.340541 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260926 12:28:59.341931 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.326891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:49655 (local address 127.7.99.62:43313)
0926 12:28:59.327019 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.327023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.328011 (+ 988us) server_negotiation.cc:407] Connection header received
0926 12:28:59.329159 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.329165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.329225 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.329312 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.331072 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.331993 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.333918 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.334129 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.334208 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.337177 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.337181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.337184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.337210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.338733 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.339162 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.339164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.339165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.339234 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.339515 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.339518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.339519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.339732 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.339833 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.341732 (+ 1899us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.341828 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:28:59.343195 15441 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43313
I20260926 12:28:59.343339 15441 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:59.343881 15441 heartbeater.cc:511] Master 127.7.99.62:43313 requested a full tablet report, sending...
I20260926 12:28:59.345043 15098 ts_manager.cc:194] Registered new tserver with Master: 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:28:59.345872 15098 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:49655
WARNING: no policy specified for oryx/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:28:59.353519 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.345258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:54331 (local address 127.7.99.62:43313)
0926 12:28:59.345392 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.345398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.345662 (+ 264us) server_negotiation.cc:407] Connection header received
0926 12:28:59.345774 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.345777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.345823 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.345918 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.346393 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.347128 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.348639 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.348910 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.349024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.350281 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.350284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.350286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.350307 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.351785 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.352214 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.352218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.352220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.352324 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.352638 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.352641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.352642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.352857 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.352983 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.353243 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.353357 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:59.353585 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.345268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.2:54331)
0926 12:28:59.345579 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.345590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.345699 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.345931 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.345933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.346149 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.346273 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.346280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.347273 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.347276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.348397 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.348410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.349056 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.349062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.349184 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.349660 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.349679 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.350145 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.351883 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.351885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.351887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.352076 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.352429 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.352439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.352441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.352531 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.352973 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.352976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.353076 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.353181 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.353358 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
I20260926 12:28:59.371114 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43313
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:28:59.479597 15450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:28:59.479837 15450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:28:59.479879 15450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:28:59.483881 15450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:28:59.483932 15450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:28:59.483969 15450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:28:59.484004 15450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:28:59.489048 15450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.99.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:28:59.489919 15450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:28:59.490892 15450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:28:59.500201 15458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:59.500284 15456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:28:59.500243 15455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:28:59.500396 15450 server_base.cc:1034] running on GCE node
I20260926 12:28:59.500761 15450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:28:59.501354 15450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:28:59.502532 15450 hybrid_clock.cc:648] HybridClock initialized: now 1790425739502524 us; error 29 us; skew 500 ppm
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:28:59.505223 15450 init.cc:377] Logged in from keytab as oryx/127.7.99.3@KRBTEST.COM (short username oryx)
I20260926 12:28:59.505995 15450 webserver.cc:460] Webserver started at http://127.7.99.3:38615/ using document root <none> and password file <none>
I20260926 12:28:59.506518 15450 fs_manager.cc:359] Metadata directory not provided
I20260926 12:28:59.506606 15450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:28:59.506832 15450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:28:59.508828 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "03698c5d081445ce8e7977ddee069216"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "4e7c96932367b6920b6a62d34fa14755"
server_key_iv: "dcb9f4a61fa6bd7bdc94df9a7a839f5c"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.509369 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "03698c5d081445ce8e7977ddee069216"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "4e7c96932367b6920b6a62d34fa14755"
server_key_iv: "dcb9f4a61fa6bd7bdc94df9a7a839f5c"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.513362 15450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:28:59.515626 15465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.516762 15450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:28:59.516894 15450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "03698c5d081445ce8e7977ddee069216"
format_stamp: "Formatted at 2026-09-26 12:28:59 on dist-test-slave-3jw2"
server_key: "4e7c96932367b6920b6a62d34fa14755"
server_key_iv: "dcb9f4a61fa6bd7bdc94df9a7a839f5c"
server_key_version: "encryptionkey@0"
I20260926 12:28:59.517004 15450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:28:59.528752 15450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:28:59.531379 15450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:28:59.531562 15450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:28:59.533130 15450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:28:59.533196 15450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:28:59.533260 15450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:28:59.533308 15450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:59.543608 15450 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:32885
I20260926 12:28:59.543637 15580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:32885 every 8 connection(s)
I20260926 12:28:59.544565 15450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:28:59.546810 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15450
I20260926 12:28:59.546911 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:28:59.546977 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.533911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:33213 (local address 127.7.99.62:43313)
0926 12:28:59.534077 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.534084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.534838 (+ 754us) server_negotiation.cc:407] Connection header received
0926 12:28:59.535971 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.535978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.536044 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.536151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.537685 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.538579 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.539876 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.540278 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.540406 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.543263 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.543268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.543273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.543304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.544996 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.545564 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.545567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.545568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.545655 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.545940 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.545942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.545943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.546126 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.546198 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.546659 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.546791 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:28:59.547266 7564 external_mini_cluster.cc:1428] Setting key 6456bcb9094d9cb8214048f9658b6d7f
I20260926 12:28:59.547742 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.534260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.3:33213)
0926 12:28:59.534682 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.534729 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.535733 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.536363 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.536374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.536866 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.537516 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.537539 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.538712 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.538717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.539752 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.539761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.540446 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.540450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.540626 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.541201 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.541216 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.543098 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.545121 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.545126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.545164 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.545458 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.545737 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.545739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.545741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.545856 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.546209 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.546214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.546421 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.546588 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.546821 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260926 12:28:59.549019 15581 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43313
I20260926 12:28:59.549157 15581 heartbeater.cc:464] Registering TS with master...
I20260926 12:28:59.549744 15581 heartbeater.cc:511] Master 127.7.99.62:43313 requested a full tablet report, sending...
I20260926 12:28:59.550679 15097 ts_manager.cc:194] Registered new tserver with Master: 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885)
I20260926 12:28:59.551441 15097 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:33213
I20260926 12:28:59.557171 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.550942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.7.99.3:58193)
0926 12:28:59.551237 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.551251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.551335 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.551554 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.551556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.551771 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.551884 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.551891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.552635 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.552639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.553444 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.553452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.553828 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.553830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.553899 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.554366 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.554378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.554682 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.555988 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.555990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.555992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.556207 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.556424 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.556426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.556427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.556508 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.556800 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.556802 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.556872 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.556947 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.557040 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260926 12:28:59.557176 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.551006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:58193 (local address 127.7.99.62:43313)
0926 12:28:59.551110 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.551113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.551300 (+ 187us) server_negotiation.cc:407] Connection header received
0926 12:28:59.551408 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.551410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.551447 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.551507 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.551988 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.552517 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.553548 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.553743 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.553810 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.554795 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.554797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.554799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.554815 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.555899 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.556282 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.556284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.556285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.556351 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.556582 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.556584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.556585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.556725 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.556796 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.557001 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.557080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:28:59.560977 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:28:59.583527 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:28:59.577368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.7.99.62:43313)
0926 12:28:59.577496 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.577499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.577510 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:28:59.577545 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.577547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.577581 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.577671 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.578439 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.578970 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.579900 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.580114 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.580312 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.583361 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37290: BlockingRecv error: recv got EOF from 127.0.0.1:37290 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:59.592032 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.584844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.7.99.62:43313)
0926 12:28:59.584948 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.584951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.585121 (+ 170us) server_negotiation.cc:407] Connection header received
0926 12:28:59.585206 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.585208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.585242 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.585306 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.585834 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.586284 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.586995 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.587187 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.587256 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.589494 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.589497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.589499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.589517 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.590691 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.591038 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.591040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.591041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.591106 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.591315 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.591317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.591318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.591476 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.591575 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.591749 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.591831 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:28:59.594957 15097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:28:59.598148 15097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:59.612085 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.0.0.1:50588)
0926 12:28:59.607119 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.607167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.607536 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.608071 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.608074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.608093 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.608233 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.608240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.609279 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.609286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.610489 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.610498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.611571 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.611574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.611682 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.611718 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.611827 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.611933 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":45,"thread_start_us":88,"threads_started":1}
I20260926 12:28:59.612119 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.7.99.3:32885)
0926 12:28:59.606804 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.606817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.607254 (+ 437us) server_negotiation.cc:407] Connection header received
0926 12:28:59.607668 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.607672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.607813 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.607984 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.608348 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.609173 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.610625 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.611443 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.611567 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.611742 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.611872 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.611967 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":96,"threads_started":1}
I20260926 12:28:59.612331 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45741 (local address 127.0.0.1:56338)
0926 12:28:59.607119 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.607167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.607288 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.607755 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.607762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.607783 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.607905 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.607915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.609381 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.609384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.610723 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.610733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.611884 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.611888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.611997 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.612020 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.612087 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.612184 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":44,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260926 12:28:59.612670 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.7.99.1:45741)
0926 12:28:59.607347 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.607361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.607389 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:28:59.607448 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.607451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.607600 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.607755 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.608045 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.609218 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.610846 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.611582 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.611673 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.611824 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.612406 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.612506 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":85,"threads_started":1}
I20260926 12:28:59.612891 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40339 (local address 127.0.0.1:48398)
0926 12:28:59.607380 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.607394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.607472 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.608506 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.608509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.608528 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.608618 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.608624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.610112 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.610115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.611237 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.611246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.612470 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.612475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.612589 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.612625 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.612699 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.612779 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":108,"threads_started":1}
I20260926 12:28:59.613531 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.606834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.7.99.2:40339)
0926 12:28:59.608065 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.608078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.608100 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:28:59.608193 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.608197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.608371 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.608513 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.608732 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.609977 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.611376 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.612321 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.613118 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.613267 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.613315 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.613388 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":120,"threads_started":1}
I20260926 12:28:59.615406 15515 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd569eb671456d80839a12bdd223a2 (DEFAULT_TABLE table=test-table [id=eb12f12ae18142e6b1925927dc0e69c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.615406 15235 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd569eb671456d80839a12bdd223a2 (DEFAULT_TABLE table=test-table [id=eb12f12ae18142e6b1925927dc0e69c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.615574 15375 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd569eb671456d80839a12bdd223a2 (DEFAULT_TABLE table=test-table [id=eb12f12ae18142e6b1925927dc0e69c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.616401 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cd569eb671456d80839a12bdd223a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.616408 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cd569eb671456d80839a12bdd223a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.616412 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cd569eb671456d80839a12bdd223a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.623149 15609 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Bootstrap starting.
I20260926 12:28:59.624518 15610 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Bootstrap starting.
I20260926 12:28:59.624518 15611 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Bootstrap starting.
I20260926 12:28:59.625268 15609 tablet_bootstrap.cc:654] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.625968 15609 log.cc:824] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:59.626494 15610 tablet_bootstrap.cc:654] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.626493 15611 tablet_bootstrap.cc:654] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.627285 15610 log.cc:824] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:59.627285 15611 log.cc:824] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:28:59.627779 15609 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: No bootstrap required, opened a new log
I20260926 12:28:59.627960 15609 ts_tablet_manager.cc:1397] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:28:59.629076 15611 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: No bootstrap required, opened a new log
I20260926 12:28:59.629076 15610 tablet_bootstrap.cc:492] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: No bootstrap required, opened a new log
I20260926 12:28:59.629242 15610 ts_tablet_manager.cc:1397] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:59.629242 15611 ts_tablet_manager.cc:1397] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:28:59.630963 15609 raft_consensus.cc:348] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.631233 15609 raft_consensus.cc:374] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.631305 15609 raft_consensus.cc:729] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.631743 15609 consensus_queue.cc:260] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.632781 15609 ts_tablet_manager.cc:1428] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 12:28:59.633131 15441 heartbeater.cc:503] Master 127.7.99.62:43313 was elected leader, sending a full tablet report...
I20260926 12:28:59.633054 15611 raft_consensus.cc:348] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.633277 15611 raft_consensus.cc:374] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.633322 15611 raft_consensus.cc:729] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.633201 15610 raft_consensus.cc:348] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.633426 15610 raft_consensus.cc:374] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.633479 15610 raft_consensus.cc:729] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.633715 15611 consensus_queue.cc:260] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.633791 15610 consensus_queue.cc:260] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.634618 15611 ts_tablet_manager.cc:1428] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:28:59.634618 15610 ts_tablet_manager.cc:1428] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:28:59.635799 15301 heartbeater.cc:503] Master 127.7.99.62:43313 was elected leader, sending a full tablet report...
I20260926 12:28:59.636363 15581 heartbeater.cc:503] Master 127.7.99.62:43313 was elected leader, sending a full tablet report...
I20260926 12:28:59.681933 15616 raft_consensus.cc:482] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:59.682104 15616 raft_consensus.cc:504] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.683182 15616 leader_election.cc:288] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:28:59.687021 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.683463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:33327 (local address 127.7.99.3:32885)
0926 12:28:59.683720 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.683725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.683741 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:28:59.683903 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.683907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.683963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.684050 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.684354 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.684962 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686050 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.686441 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686556 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.686617 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.686825 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.686893 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260926 12:28:59.687043 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.683379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.7.99.1:33327)
0926 12:28:59.683624 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.683643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.683743 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.684064 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.684068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.684085 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.684250 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.684256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.685068 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.685075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.685933 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.685943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.686559 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.686703 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.686722 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.686811 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.686885 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":46,"thread_start_us":69,"threads_started":1}
I20260926 12:28:59.687142 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.683631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42763 (local address 127.7.99.2:40339)
0926 12:28:59.683764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.683768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.684019 (+ 251us) server_negotiation.cc:407] Connection header received
0926 12:28:59.684176 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.684178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.684248 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.684329 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.684591 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.685083 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686118 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.686506 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686618 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.686688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.686921 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.687003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:28:59.687343 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.683413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40339 (local address 127.7.99.1:42763)
0926 12:28:59.683920 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.683933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.684055 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.684342 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.684345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.684361 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.684476 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.684481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.685196 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.685200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.685994 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.686005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.686634 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.686637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.686800 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.686817 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.686905 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.686974 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260926 12:28:59.687630 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cd569eb671456d80839a12bdd223a2" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216" is_pre_election: true
I20260926 12:28:59.688006 15535 raft_consensus.cc:2454] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 0.
I20260926 12:28:59.688169 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cd569eb671456d80839a12bdd223a2" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c" is_pre_election: true
I20260926 12:28:59.688455 15395 raft_consensus.cc:2454] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 0.
I20260926 12:28:59.688532 15188 leader_election.cc:302] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8; no voters:
I20260926 12:28:59.688892 15616 raft_consensus.cc:2781] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:59.689002 15616 raft_consensus.cc:482] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:59.689061 15616 raft_consensus.cc:3037] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.690090 15616 raft_consensus.cc:504] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.690449 15616 leader_election.cc:288] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 election: Requested vote from peers 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:28:59.690810 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cd569eb671456d80839a12bdd223a2" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216"
I20260926 12:28:59.690810 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cd569eb671456d80839a12bdd223a2" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c"
I20260926 12:28:59.690932 15395 raft_consensus.cc:3037] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.690932 15535 raft_consensus.cc:3037] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.692296 15395 raft_consensus.cc:2454] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 1.
I20260926 12:28:59.692289 15535 raft_consensus.cc:2454] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 1.
I20260926 12:28:59.692593 15189 leader_election.cc:302] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8, 4fe97be3f2564511bb49824c1c1c151c; no voters:
I20260926 12:28:59.692799 15616 raft_consensus.cc:2781] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:59.692981 15616 raft_consensus.cc:686] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Running, Role: LEADER
I20260926 12:28:59.693306 15616 consensus_queue.cc:237] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } }
I20260926 12:28:59.696408 15097 catalog_manager.cc:5495] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } health_report { overall_health: UNKNOWN } } }
I20260926 12:28:59.707774 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.704719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.7.99.1:45741)
0926 12:28:59.704831 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.704835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.704847 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:28:59.704960 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.704962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.705011 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.705093 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:28:59.705286 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.705772 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.706555 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.706766 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.706840 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.707050 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:28:59.707471 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:28:59.707553 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.707635 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.707680 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:28:59.713546 15395 raft_consensus.cc:1261] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Refusing update from remote peer 3e2af0a3f0f54be3804ecb992d66cfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:59.713546 15535 raft_consensus.cc:1261] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Refusing update from remote peer 3e2af0a3f0f54be3804ecb992d66cfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:28:59.714198 15620 consensus_queue.cc:1035] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:59.714421 15616 consensus_queue.cc:1035] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:28:59.722179 15625 mvcc.cc:205] Tried to move back new op lower bound from 7333583829858914304 to 7333583829786673152. Current Snapshot: MvccSnapshot[applied={T|T < 7333583829858914304}]
I20260926 12:28:59.724086 15624 mvcc.cc:205] Tried to move back new op lower bound from 7333583829858914304 to 7333583829786673152. Current Snapshot: MvccSnapshot[applied={T|T < 7333583829858914304}]
I20260926 12:28:59.728307 15097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:59.728447 15097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:28:59.730679 15097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Sending DeleteTablet for 3 replicas of tablet 65cd569eb671456d80839a12bdd223a2
I20260926 12:28:59.731357 15515 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cd569eb671456d80839a12bdd223a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:59 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50588
I20260926 12:28:59.731498 15375 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cd569eb671456d80839a12bdd223a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:59 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48398
I20260926 12:28:59.731526 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cd569eb671456d80839a12bdd223a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:28:59 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:56338
I20260926 12:28:59.731940 15634 tablet_replica.cc:330] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: stopping tablet replica
I20260926 12:28:59.732242 15634 raft_consensus.cc:2229] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:28:59.732630 15634 raft_consensus.cc:2258] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:59.732656 15633 tablet_replica.cc:330] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: stopping tablet replica
I20260926 12:28:59.732756 15097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:28:59.732911 15633 raft_consensus.cc:2229] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:59.733093 15635 tablet_replica.cc:330] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: stopping tablet replica
I20260926 12:28:59.733179 15633 raft_consensus.cc:2258] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:28:59.733179 15097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:59.733340 15635 raft_consensus.cc:2229] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:28:59.733587 15635 raft_consensus.cc:2258] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:28:59.734587 15634 ts_tablet_manager.cc:1905] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:59.734587 15633 ts_tablet_manager.cc:1905] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:59.737200 15633 ts_tablet_manager.cc:1918] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:59.737200 15634 ts_tablet_manager.cc:1918] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:59.737291 15633 log.cc:1196] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/65cd569eb671456d80839a12bdd223a2
I20260926 12:28:59.737293 15634 log.cc:1196] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/65cd569eb671456d80839a12bdd223a2
I20260926 12:28:59.737603 15633 ts_tablet_manager.cc:1939] T 65cd569eb671456d80839a12bdd223a2 P 03698c5d081445ce8e7977ddee069216: Deleting consensus metadata
I20260926 12:28:59.737620 15634 ts_tablet_manager.cc:1939] T 65cd569eb671456d80839a12bdd223a2 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting consensus metadata
I20260926 12:28:59.738669 15084 catalog_manager.cc:4841] TS 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885): tablet 65cd569eb671456d80839a12bdd223a2 (table test-table [id=eb12f12ae18142e6b1925927dc0e69c6]) successfully deleted
I20260926 12:28:59.739374 15082 catalog_manager.cc:4841] TS 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741): tablet 65cd569eb671456d80839a12bdd223a2 (table test-table [id=eb12f12ae18142e6b1925927dc0e69c6]) successfully deleted
I20260926 12:28:59.739377 15515 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f (DEFAULT_TABLE table=test-table [id=83484a603e2f49d385f7dd5809876198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.739612 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20f5539e3a4b1f9782b94760a90c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.739630 15235 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f (DEFAULT_TABLE table=test-table [id=83484a603e2f49d385f7dd5809876198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.739869 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20f5539e3a4b1f9782b94760a90c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.740864 15375 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f (DEFAULT_TABLE table=test-table [id=83484a603e2f49d385f7dd5809876198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:28:59.741142 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20f5539e3a4b1f9782b94760a90c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.741849 15610 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Bootstrap starting.
I20260926 12:28:59.742092 15611 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Bootstrap starting.
I20260926 12:28:59.742761 15610 tablet_bootstrap.cc:654] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.743036 15611 tablet_bootstrap.cc:654] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.743727 15609 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Bootstrap starting.
I20260926 12:28:59.744521 15611 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: No bootstrap required, opened a new log
I20260926 12:28:59.744607 15611 ts_tablet_manager.cc:1397] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:28:59.744630 15609 tablet_bootstrap.cc:654] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.745014 15635 ts_tablet_manager.cc:1905] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:28:59.745018 15611 raft_consensus.cc:348] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.745134 15611 raft_consensus.cc:374] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.745177 15611 raft_consensus.cc:729] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.745306 15611 consensus_queue.cc:260] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.745589 15611 ts_tablet_manager.cc:1428] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:59.746342 15610 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: No bootstrap required, opened a new log
I20260926 12:28:59.746425 15610 ts_tablet_manager.cc:1397] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 12:28:59.746833 15610 raft_consensus.cc:348] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.746937 15610 raft_consensus.cc:374] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.746974 15610 raft_consensus.cc:729] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.747092 15610 consensus_queue.cc:260] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.747332 15610 ts_tablet_manager.cc:1428] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:59.747843 15635 ts_tablet_manager.cc:1918] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:28:59.747959 15635 log.cc:1196] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/65cd569eb671456d80839a12bdd223a2
I20260926 12:28:59.748322 15635 ts_tablet_manager.cc:1939] T 65cd569eb671456d80839a12bdd223a2 P 4fe97be3f2564511bb49824c1c1c151c: Deleting consensus metadata
I20260926 12:28:59.749462 15609 tablet_bootstrap.cc:492] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: No bootstrap required, opened a new log
I20260926 12:28:59.749541 15609 ts_tablet_manager.cc:1397] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 12:28:59.749504 15085 catalog_manager.cc:4841] TS 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339): tablet 65cd569eb671456d80839a12bdd223a2 (table test-table [id=eb12f12ae18142e6b1925927dc0e69c6]) successfully deleted
I20260926 12:28:59.749936 15609 raft_consensus.cc:348] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.750046 15609 raft_consensus.cc:374] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.750077 15609 raft_consensus.cc:729] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.750190 15609 consensus_queue.cc:260] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.750427 15609 ts_tablet_manager.cc:1428] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 12:28:59.795578 15582 tablet.cc:2369] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:28:59.839442 15442 tablet.cc:2369] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:28:59.879468 15302 tablet.cc:2369] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:28:59.912249 15615 raft_consensus.cc:482] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:28:59.912424 15615 raft_consensus.cc:504] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.913647 15615 leader_election.cc:288] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885)
I20260926 12:28:59.918810 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.914226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:33923 (local address 127.7.99.1:45741)
0926 12:28:59.914387 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.914391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.914404 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:28:59.914460 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.914463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.914504 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.914577 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.914972 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.915743 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.917511 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.918107 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.918216 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.918280 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.918611 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.918681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:28:59.918810 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.914015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:54781 (local address 127.7.99.3:32885)
0926 12:28:59.914152 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.914156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.914280 (+ 124us) server_negotiation.cc:407] Connection header received
0926 12:28:59.914434 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.914439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.914489 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.914577 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:28:59.914972 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.915743 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.917511 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.918107 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.918216 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.918278 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.918611 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.918681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:28:59.918887 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.913887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45741 (local address 127.7.99.2:33923)
0926 12:28:59.914264 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.914276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.914336 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.914608 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.914611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.914651 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.914850 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.914858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.915855 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.915858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.917387 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.917398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.918232 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.918235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.918422 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.918500 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.918580 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.918691 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":252,"spinlock_wait_cycles":6272,"thread_start_us":73,"threads_started":1}
I20260926 12:28:59.918890 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.913887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.7.99.2:54781)
0926 12:28:59.914176 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.914191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.914314 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.914608 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.914611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.914650 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:28:59.914851 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.914858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.915855 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.915858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.917387 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.917398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.918232 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.918235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.918444 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.918514 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.918580 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.918694 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":327,"spinlock_wait_cycles":28032,"thread_start_us":103,"threads_started":1}
I20260926 12:28:59.919461 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20f5539e3a4b1f9782b94760a90c4f" candidate_uuid: "4fe97be3f2564511bb49824c1c1c151c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216" is_pre_election: true
I20260926 12:28:59.919486 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20f5539e3a4b1f9782b94760a90c4f" candidate_uuid: "4fe97be3f2564511bb49824c1c1c151c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" is_pre_election: true
I20260926 12:28:59.919624 15535 raft_consensus.cc:2454] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe97be3f2564511bb49824c1c1c151c in term 0.
I20260926 12:28:59.919706 15255 raft_consensus.cc:2454] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe97be3f2564511bb49824c1c1c151c in term 0.
I20260926 12:28:59.920027 15328 leader_election.cc:302] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 4fe97be3f2564511bb49824c1c1c151c; no voters:
I20260926 12:28:59.920348 15615 raft_consensus.cc:2781] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:28:59.920414 15615 raft_consensus.cc:482] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:28:59.920452 15615 raft_consensus.cc:3037] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.921545 15615 raft_consensus.cc:504] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.921886 15615 leader_election.cc:288] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [CANDIDATE]: Term 1 election: Requested vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885)
I20260926 12:28:59.922204 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20f5539e3a4b1f9782b94760a90c4f" candidate_uuid: "4fe97be3f2564511bb49824c1c1c151c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
I20260926 12:28:59.922327 15255 raft_consensus.cc:3037] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.922290 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20f5539e3a4b1f9782b94760a90c4f" candidate_uuid: "4fe97be3f2564511bb49824c1c1c151c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216"
I20260926 12:28:59.922433 15535 raft_consensus.cc:3037] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:28:59.923247 15255 raft_consensus.cc:2454] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe97be3f2564511bb49824c1c1c151c in term 1.
I20260926 12:28:59.923336 15535 raft_consensus.cc:2454] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe97be3f2564511bb49824c1c1c151c in term 1.
I20260926 12:28:59.923622 15328 leader_election.cc:302] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 4fe97be3f2564511bb49824c1c1c151c; no voters:
I20260926 12:28:59.923785 15615 raft_consensus.cc:2781] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:28:59.923990 15615 raft_consensus.cc:686] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 1 LEADER]: Becoming Leader. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Running, Role: LEADER
I20260926 12:28:59.924283 15615 consensus_queue.cc:237] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.926391 15098 catalog_manager.cc:5495] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2). New cstate: current_term: 1 leader_uuid: "4fe97be3f2564511bb49824c1c1c151c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:28:59 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425738, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.62@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:28:59.969439 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.961851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43313 (local address 127.0.0.1:37302)
0926 12:28:59.962147 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:28:59.962161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:28:59.962262 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:28:59.962487 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:28:59.962489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:28:59.962877 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:28:59.962968 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.962973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.963639 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.963642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.964252 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:28:59.964260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.964648 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.964650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:28:59.964707 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.965245 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:28:59.965262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:28:59.966679 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:28:59.968056 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.968064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.968068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.968348 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.968588 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:28:59.968590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:28:59.968591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:28:59.968666 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:28:59.968978 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:28:59.968983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:28:59.969081 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 12:28:59.969170 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:28:59.969278 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":129,"threads_started":1}
I20260926 12:28:59.969453 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:28:59.961890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.7.99.62:43313)
0926 12:28:59.962036 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:28:59.962040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:28:59.962236 (+ 196us) server_negotiation.cc:407] Connection header received
0926 12:28:59.962339 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:28:59.962341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:28:59.962381 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:28:59.962451 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:28:59.963063 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.963554 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.964353 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:28:59.964551 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:28:59.964624 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:28:59.966798 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:28:59.966801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:28:59.966803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:28:59.966824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:28:59.967965 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.968428 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.968431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.968432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.968509 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:28:59.968736 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:28:59.968738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:28:59.968739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:28:59.968886 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:28:59.968953 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:28:59.969223 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 12:28:59.969331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":63}
I20260926 12:28:59.973081 15098 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:37302:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:28:59.973788 15098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:28:59.978442 15235 tablet_service.cc:1467] Processing CreateTablet for tablet 559b908d82d4461ebfb33a8c25c50dfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71a4118d13444c56a6701bd76a04a22a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:59.978453 15375 tablet_service.cc:1467] Processing CreateTablet for tablet 559b908d82d4461ebfb33a8c25c50dfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71a4118d13444c56a6701bd76a04a22a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:59.978617 15515 tablet_service.cc:1467] Processing CreateTablet for tablet 559b908d82d4461ebfb33a8c25c50dfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71a4118d13444c56a6701bd76a04a22a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:28:59.978845 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559b908d82d4461ebfb33a8c25c50dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.978845 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559b908d82d4461ebfb33a8c25c50dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.978952 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559b908d82d4461ebfb33a8c25c50dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:28:59.981662 15611 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: Bootstrap starting.
I20260926 12:28:59.982405 15611 tablet_bootstrap.cc:654] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.982563 15610 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: Bootstrap starting.
I20260926 12:28:59.983153 15609 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: Bootstrap starting.
I20260926 12:28:59.983510 15610 tablet_bootstrap.cc:654] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.983613 15611 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: No bootstrap required, opened a new log
I20260926 12:28:59.983695 15611 ts_tablet_manager.cc:1397] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 12:28:59.984071 15611 raft_consensus.cc:348] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.984200 15609 tablet_bootstrap.cc:654] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: Neither blocks nor log segments found. Creating new log.
I20260926 12:28:59.984236 15611 raft_consensus.cc:374] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.984282 15611 raft_consensus.cc:729] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.984412 15611 consensus_queue.cc:260] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.984689 15611 ts_tablet_manager.cc:1428] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:28:59.984786 15610 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: No bootstrap required, opened a new log
I20260926 12:28:59.984862 15610 ts_tablet_manager.cc:1397] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:59.984922 15620 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:28:59.985037 15620 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:59.985312 15610 raft_consensus.cc:348] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.985410 15610 raft_consensus.cc:374] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.985440 15610 raft_consensus.cc:729] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.985491 15609 tablet_bootstrap.cc:492] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: No bootstrap required, opened a new log
I20260926 12:28:59.985571 15609 ts_tablet_manager.cc:1397] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:28:59.985553 15610 consensus_queue.cc:260] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.985790 15610 ts_tablet_manager.cc:1428] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:59.985980 15609 raft_consensus.cc:348] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.986089 15609 raft_consensus.cc:374] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:28:59.986037 15617 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:28:59.986125 15609 raft_consensus.cc:729] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Initialized, Role: FOLLOWER
I20260926 12:28:59.986173 15617 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:28:59.986235 15609 consensus_queue.cc:260] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:28:59.986502 15609 ts_tablet_manager.cc:1428] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:28:59.986515 15615 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:28:59.987021 15615 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:29:00.201146 15617 raft_consensus.cc:482] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:29:00.201292 15617 raft_consensus.cc:504] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.202448 15617 leader_election.cc:288] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:29:00.207882 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.202729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:33983 (local address 127.7.99.1:45741)
0926 12:29:00.202853 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.202859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.203107 (+ 248us) server_negotiation.cc:407] Connection header received
0926 12:29:00.203251 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.203254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.203306 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.203394 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:00.203820 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.204651 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.206495 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.207095 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.207204 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.207268 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.207677 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.207749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:29:00.207883 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.202813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:56675 (local address 127.7.99.2:40339)
0926 12:29:00.203104 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.203108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.203125 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:29:00.203251 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.203254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.203305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.203393 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:00.203820 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.204650 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.206495 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.207095 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.207204 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.207269 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.207677 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.207749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260926 12:29:00.207933 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.202743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40339 (local address 127.7.99.3:56675)
0926 12:29:00.202964 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.203010 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.203133 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.203417 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.203420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.203444 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:00.203700 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.203708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.204774 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.204778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.206368 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.206380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.207216 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.207219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.207533 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.207573 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.207644 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.207765 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":409,"spinlock_wait_cycles":26496,"thread_start_us":71,"threads_started":1}
I20260926 12:29:00.208159 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.202666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45741 (local address 127.7.99.3:33983)
0926 12:29:00.202964 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.202997 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.203133 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.203417 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.203420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.203444 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:00.203704 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.203709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.204775 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.204778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.206368 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.206380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.207216 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.207219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.207512 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.207564 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.207644 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.207765 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":322,"spinlock_wait_cycles":15744,"thread_start_us":69,"threads_started":1}
I20260926 12:29:00.208521 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559b908d82d4461ebfb33a8c25c50dfa" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c" is_pre_election: true
I20260926 12:29:00.208578 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559b908d82d4461ebfb33a8c25c50dfa" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" is_pre_election: true
I20260926 12:29:00.208686 15395 raft_consensus.cc:2454] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 0.
I20260926 12:29:00.208737 15255 raft_consensus.cc:2454] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 0.
I20260926 12:29:00.209116 15466 leader_election.cc:302] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8; no voters:
I20260926 12:29:00.209358 15617 raft_consensus.cc:2781] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:29:00.209416 15617 raft_consensus.cc:482] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:29:00.209455 15617 raft_consensus.cc:3037] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.210386 15617 raft_consensus.cc:504] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.210736 15617 leader_election.cc:288] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:29:00.211030 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559b908d82d4461ebfb33a8c25c50dfa" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
I20260926 12:29:00.211087 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559b908d82d4461ebfb33a8c25c50dfa" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c"
I20260926 12:29:00.211148 15255 raft_consensus.cc:3037] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.211212 15395 raft_consensus.cc:3037] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.212095 15255 raft_consensus.cc:2454] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 1.
I20260926 12:29:00.212215 15395 raft_consensus.cc:2454] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 1.
I20260926 12:29:00.212422 15466 leader_election.cc:302] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8; no voters:
I20260926 12:29:00.212587 15617 raft_consensus.cc:2781] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:00.212739 15617 raft_consensus.cc:686] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [term 1 LEADER]: Becoming Leader. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Running, Role: LEADER
I20260926 12:29:00.213001 15617 consensus_queue.cc:237] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.214095 15655 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: TxnStatusTablet state changed. Reason: New leader 03698c5d081445ce8e7977ddee069216. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.214232 15655 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:29:00.214752 15657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:29:00.214921 15657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:29:00.215337 15097 catalog_manager.cc:5495] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03698c5d081445ce8e7977ddee069216 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } health_report { overall_health: HEALTHY } } }
I20260926 12:29:00.282214 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.278360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.0.0.1:50600)
0926 12:29:00.278532 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.278549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.278650 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.278959 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.278961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.278969 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:29:00.279055 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.279059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.279842 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.279847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.280648 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.280658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.281119 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.281123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.281214 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.281254 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.281907 (+ 653us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.281921 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.281992 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.282080 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 12:29:00.282200 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.278514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.7.99.3:32885)
0926 12:29:00.278661 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.278667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.278683 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:29:00.278759 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.278762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.278825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.278938 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.279184 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.279739 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.280764 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.281019 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.281122 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.281394 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.281812 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.281943 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.282004 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.282055 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:29:00.285012 15395 raft_consensus.cc:1261] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Refusing update from remote peer 03698c5d081445ce8e7977ddee069216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.285272 15255 raft_consensus.cc:1261] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Refusing update from remote peer 03698c5d081445ce8e7977ddee069216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.285457 15617 consensus_queue.cc:1035] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.285764 15655 consensus_queue.cc:1035] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.286882 15615 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: TxnStatusTablet state changed. Reason: New leader 03698c5d081445ce8e7977ddee069216. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.287001 15615 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:29:00.287456 15620 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: TxnStatusTablet state changed. Reason: New leader 03698c5d081445ce8e7977ddee069216. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.287569 15620 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:29:00.289108 15655 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.289234 15655 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:29:00.290016 15617 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.290112 15617 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 03698c5d081445ce8e7977ddee069216: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:29:00.290127 15620 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.290234 15620 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 3e2af0a3f0f54be3804ecb992d66cfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:29:00.290867 15615 tablet_replica.cc:439] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } } }
I20260926 12:29:00.290964 15615 tablet_replica.cc:442] T 559b908d82d4461ebfb33a8c25c50dfa P 4fe97be3f2564511bb49824c1c1c151c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:29:00.291721 15663 mvcc.cc:205] Tried to move back new op lower bound from 7333583832202334208 to 7333583831914696704. Current Snapshot: MvccSnapshot[applied={T|T < 7333583832202334208}]
I20260926 12:29:00.300423 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.296481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.7.99.2:40339)
0926 12:29:00.296583 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.296587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.296788 (+ 201us) server_negotiation.cc:407] Connection header received
0926 12:29:00.296948 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.296950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.297000 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.297081 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.297326 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.297873 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.299118 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.299343 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.299421 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.299689 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.300084 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.300204 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.300246 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.300290 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:29:00.311035 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.307387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:53903 (local address 127.7.99.3:32885)
0926 12:29:00.307528 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.307533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.307695 (+ 162us) server_negotiation.cc:407] Connection header received
0926 12:29:00.307811 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.307813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.307853 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.307912 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.308222 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.308798 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.309968 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.310166 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.310237 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.310555 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.310617 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.310668 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.310815 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.310891 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:29:00.311046 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.307311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.7.99.2:53903)
0926 12:29:00.307610 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.307624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.307725 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.307948 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.307951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.307959 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:29:00.308077 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.308083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.308911 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.308914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.309840 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.309854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.310276 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.310280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.310373 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.310422 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.310701 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.310708 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.310800 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.310880 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260926 12:29:00.313104 15615 consensus_queue.cc:1035] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.320206 15641 consensus_queue.cc:1035] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.344856 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.341191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:37399 (local address 127.7.99.2:40339)
0926 12:29:00.341308 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.341312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.341327 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:29:00.341372 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.341377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.341424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.341501 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.341828 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.342339 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.343597 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.343909 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.343984 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.344440 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.344521 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.344603 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.344687 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.344730 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:29:00.344905 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.340469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40339 (local address 127.7.99.3:37399)
0926 12:29:00.340766 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.340778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.340851 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.341616 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.341618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.341625 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:29:00.341723 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.341727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.342456 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.342459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.343469 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.343479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.344108 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.344111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.344234 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.344289 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.344606 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.344614 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.344678 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.344748 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":116,"threads_started":1}
I20260926 12:29:00.366248 15098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.366421 15098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.368685 15098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Sending DeleteTablet for 3 replicas of tablet 9a20f5539e3a4b1f9782b94760a90c4f
I20260926 12:29:00.369243 15375 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48398
I20260926 12:29:00.369338 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:56338
I20260926 12:29:00.373075 15515 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20f5539e3a4b1f9782b94760a90c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50588
I20260926 12:29:00.376415 15681 tablet_replica.cc:330] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: stopping tablet replica
I20260926 12:29:00.376591 15681 raft_consensus.cc:2229] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:29:00.376916 15681 raft_consensus.cc:2258] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.377099 15682 tablet_replica.cc:330] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: stopping tablet replica
I20260926 12:29:00.377230 15682 raft_consensus.cc:2229] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.377353 15682 raft_consensus.cc:2258] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.377913 15681 ts_tablet_manager.cc:1905] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.378217 15682 ts_tablet_manager.cc:1905] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.379482 15684 tablet_replica.cc:330] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: stopping tablet replica
I20260926 12:29:00.379660 15684 raft_consensus.cc:2229] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.379971 15684 raft_consensus.cc:2258] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.381474 15684 ts_tablet_manager.cc:1905] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.382879 15681 ts_tablet_manager.cc:1918] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:29:00.384057 15682 ts_tablet_manager.cc:1918] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:29:00.384150 15682 log.cc:1196] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/9a20f5539e3a4b1f9782b94760a90c4f
I20260926 12:29:00.384303 15681 log.cc:1196] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/9a20f5539e3a4b1f9782b94760a90c4f
I20260926 12:29:00.384438 15682 ts_tablet_manager.cc:1939] T 9a20f5539e3a4b1f9782b94760a90c4f P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting consensus metadata
I20260926 12:29:00.385160 15082 catalog_manager.cc:4841] TS 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741): tablet 9a20f5539e3a4b1f9782b94760a90c4f (table test-table [id=83484a603e2f49d385f7dd5809876198]) successfully deleted
I20260926 12:29:00.385474 15681 ts_tablet_manager.cc:1939] T 9a20f5539e3a4b1f9782b94760a90c4f P 4fe97be3f2564511bb49824c1c1c151c: Deleting consensus metadata
I20260926 12:29:00.386142 15085 catalog_manager.cc:4841] TS 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339): tablet 9a20f5539e3a4b1f9782b94760a90c4f (table test-table [id=83484a603e2f49d385f7dd5809876198]) successfully deleted
I20260926 12:29:00.387262 15684 ts_tablet_manager.cc:1918] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:29:00.387380 15684 log.cc:1196] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/9a20f5539e3a4b1f9782b94760a90c4f
I20260926 12:29:00.387651 15684 ts_tablet_manager.cc:1939] T 9a20f5539e3a4b1f9782b94760a90c4f P 03698c5d081445ce8e7977ddee069216: Deleting consensus metadata
I20260926 12:29:00.388309 15084 catalog_manager.cc:4841] TS 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885): tablet 9a20f5539e3a4b1f9782b94760a90c4f (table test-table [id=83484a603e2f49d385f7dd5809876198]) successfully deleted
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:29:00.404896 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:00.398074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.7.99.62:43313)
0926 12:29:00.398206 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.398210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.398429 (+ 219us) server_negotiation.cc:407] Connection header received
0926 12:29:00.398572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.398574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.398619 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.398725 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:00.399542 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.400046 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.401257 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.401464 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.401546 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.404746 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37304: BlockingRecv error: recv got EOF from 127.0.0.1:37304 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM
I20260926 12:29:00.414759 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.406153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.7.99.62:43313)
0926 12:29:00.406249 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.406253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.406369 (+ 116us) server_negotiation.cc:407] Connection header received
0926 12:29:00.406514 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.406517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.406557 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.406635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:00.407475 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.408278 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.409014 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.409360 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.409473 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.411861 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:00.411863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:00.411865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:00.411887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:00.413310 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:00.413745 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:00.413747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:00.413748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:00.413817 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:00.414075 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:00.414077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:00.414078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:00.414231 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:00.414326 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.414561 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.414648 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:29:00.417245 15098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:29:00.417690 15098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:29:00.422465 15515 tablet_service.cc:1467] Processing CreateTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 (DEFAULT_TABLE table=test-table [id=64228ceac2fc482791c7da53f9d7a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.422465 15235 tablet_service.cc:1467] Processing CreateTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 (DEFAULT_TABLE table=test-table [id=64228ceac2fc482791c7da53f9d7a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.422752 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6684c0279f34925bb7bb2f7f5d91461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.422751 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6684c0279f34925bb7bb2f7f5d91461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.425110 15610 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Bootstrap starting.
I20260926 12:29:00.425434 15375 tablet_service.cc:1467] Processing CreateTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 (DEFAULT_TABLE table=test-table [id=64228ceac2fc482791c7da53f9d7a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.425639 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6684c0279f34925bb7bb2f7f5d91461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.425915 15610 tablet_bootstrap.cc:654] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.426463 15611 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Bootstrap starting.
I20260926 12:29:00.427238 15611 tablet_bootstrap.cc:654] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.427560 15609 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Bootstrap starting.
I20260926 12:29:00.428046 15610 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: No bootstrap required, opened a new log
I20260926 12:29:00.428124 15610 ts_tablet_manager.cc:1397] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:29:00.428472 15609 tablet_bootstrap.cc:654] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.428874 15610 raft_consensus.cc:348] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.429000 15610 raft_consensus.cc:374] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.429032 15610 raft_consensus.cc:729] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.429150 15610 consensus_queue.cc:260] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.429395 15610 ts_tablet_manager.cc:1428] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.429647 15609 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: No bootstrap required, opened a new log
I20260926 12:29:00.429744 15609 ts_tablet_manager.cc:1397] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:29:00.430233 15611 tablet_bootstrap.cc:492] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: No bootstrap required, opened a new log
I20260926 12:29:00.430215 15609 raft_consensus.cc:348] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.430310 15611 ts_tablet_manager.cc:1397] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 12:29:00.430316 15609 raft_consensus.cc:374] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.430348 15609 raft_consensus.cc:729] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.430476 15609 consensus_queue.cc:260] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.430682 15611 raft_consensus.cc:348] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.430744 15609 ts_tablet_manager.cc:1428] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.430775 15611 raft_consensus.cc:374] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.430805 15611 raft_consensus.cc:729] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.430909 15611 consensus_queue.cc:260] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.431141 15611 ts_tablet_manager.cc:1428] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.455848 15617 raft_consensus.cc:482] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:29:00.455991 15617 raft_consensus.cc:504] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.456514 15617 leader_election.cc:288] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:29:00.456876 15254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6684c0279f34925bb7bb2f7f5d91461" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" is_pre_election: true
I20260926 12:29:00.457021 15254 raft_consensus.cc:2454] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 0.
I20260926 12:29:00.456961 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6684c0279f34925bb7bb2f7f5d91461" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c" is_pre_election: true
I20260926 12:29:00.457124 15395 raft_consensus.cc:2454] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 0.
I20260926 12:29:00.457319 15466 leader_election.cc:302] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8; no voters:
I20260926 12:29:00.457502 15617 raft_consensus.cc:2781] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:29:00.457556 15617 raft_consensus.cc:482] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:29:00.457602 15617 raft_consensus.cc:3037] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.458781 15617 raft_consensus.cc:504] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.459108 15617 leader_election.cc:288] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741), 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339)
I20260926 12:29:00.459432 15254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6684c0279f34925bb7bb2f7f5d91461" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8"
I20260926 12:29:00.459510 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6684c0279f34925bb7bb2f7f5d91461" candidate_uuid: "03698c5d081445ce8e7977ddee069216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c"
I20260926 12:29:00.459564 15254 raft_consensus.cc:3037] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.459611 15395 raft_consensus.cc:3037] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.460717 15254 raft_consensus.cc:2454] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 1.
I20260926 12:29:00.460717 15395 raft_consensus.cc:2454] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03698c5d081445ce8e7977ddee069216 in term 1.
I20260926 12:29:00.461002 15466 leader_election.cc:302] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03698c5d081445ce8e7977ddee069216, 3e2af0a3f0f54be3804ecb992d66cfe8; no voters:
I20260926 12:29:00.461155 15617 raft_consensus.cc:2781] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:00.461253 15617 raft_consensus.cc:686] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 1 LEADER]: Becoming Leader. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Running, Role: LEADER
I20260926 12:29:00.461406 15617 consensus_queue.cc:237] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.463002 15097 catalog_manager.cc:5495] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03698c5d081445ce8e7977ddee069216 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "03698c5d081445ce8e7977ddee069216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } health_report { overall_health: HEALTHY } } }
I20260926 12:29:00.481986 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.478292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.7.99.3:32885)
0926 12:29:00.478438 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.478443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.478459 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:29:00.478599 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.478602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.478650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.478743 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.479015 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.479773 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.480884 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.481196 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.481320 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.481531 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.481622 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.481707 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.481786 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.481835 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:29:00.484124 15395 raft_consensus.cc:1261] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Refusing update from remote peer 03698c5d081445ce8e7977ddee069216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.484422 15254 raft_consensus.cc:1261] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Refusing update from remote peer 03698c5d081445ce8e7977ddee069216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.484608 15617 consensus_queue.cc:1035] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.484953 15676 consensus_queue.cc:1035] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.495201 15098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.495368 15098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.497469 15098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Sending DeleteTablet for 3 replicas of tablet f6684c0279f34925bb7bb2f7f5d91461
I20260926 12:29:00.498211 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:56338
I20260926 12:29:00.498241 15375 tablet_service.cc:1514] Processing DeleteTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48398
I20260926 12:29:00.498386 15682 tablet_replica.cc:330] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: stopping tablet replica
I20260926 12:29:00.498421 15681 tablet_replica.cc:330] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: stopping tablet replica
I20260926 12:29:00.498515 15682 raft_consensus.cc:2229] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.498538 15681 raft_consensus.cc:2229] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.498651 15681 raft_consensus.cc:2258] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.498654 15682 raft_consensus.cc:2258] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.499245 15515 tablet_service.cc:1514] Processing DeleteTablet for tablet f6684c0279f34925bb7bb2f7f5d91461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50588
I20260926 12:29:00.499398 15684 tablet_replica.cc:330] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: stopping tablet replica
I20260926 12:29:00.499502 15684 raft_consensus.cc:2229] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:29:00.499619 15681 ts_tablet_manager.cc:1905] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.499773 15684 raft_consensus.cc:2258] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.500586 15684 ts_tablet_manager.cc:1905] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.502089 15681 ts_tablet_manager.cc:1918] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:29:00.502154 15681 log.cc:1196] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/f6684c0279f34925bb7bb2f7f5d91461
I20260926 12:29:00.502398 15681 ts_tablet_manager.cc:1939] T f6684c0279f34925bb7bb2f7f5d91461 P 4fe97be3f2564511bb49824c1c1c151c: Deleting consensus metadata
I20260926 12:29:00.503053 15684 ts_tablet_manager.cc:1918] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:29:00.503125 15684 log.cc:1196] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/f6684c0279f34925bb7bb2f7f5d91461
I20260926 12:29:00.503381 15684 ts_tablet_manager.cc:1939] T f6684c0279f34925bb7bb2f7f5d91461 P 03698c5d081445ce8e7977ddee069216: Deleting consensus metadata
I20260926 12:29:00.504082 15084 catalog_manager.cc:4841] TS 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885): tablet f6684c0279f34925bb7bb2f7f5d91461 (table test-table [id=64228ceac2fc482791c7da53f9d7a295]) successfully deleted
I20260926 12:29:00.504944 15085 catalog_manager.cc:4841] TS 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339): tablet f6684c0279f34925bb7bb2f7f5d91461 (table test-table [id=64228ceac2fc482791c7da53f9d7a295]) successfully deleted
I20260926 12:29:00.505273 15098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:29:00.505445 15682 ts_tablet_manager.cc:1905] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:29:00.505664 15098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:29:00.507823 15682 ts_tablet_manager.cc:1918] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:29:00.507891 15682 log.cc:1196] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/f6684c0279f34925bb7bb2f7f5d91461
I20260926 12:29:00.508150 15682 ts_tablet_manager.cc:1939] T f6684c0279f34925bb7bb2f7f5d91461 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting consensus metadata
I20260926 12:29:00.508987 15082 catalog_manager.cc:4841] TS 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741): tablet f6684c0279f34925bb7bb2f7f5d91461 (table test-table [id=64228ceac2fc482791c7da53f9d7a295]) successfully deleted
I20260926 12:29:00.509922 15235 tablet_service.cc:1467] Processing CreateTablet for tablet 12a4d6c496214cb78005877727536301 (DEFAULT_TABLE table=test-table [id=f271dae8300747fc95604c94cb86b0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.510175 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a4d6c496214cb78005877727536301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.510178 15375 tablet_service.cc:1467] Processing CreateTablet for tablet 12a4d6c496214cb78005877727536301 (DEFAULT_TABLE table=test-table [id=f271dae8300747fc95604c94cb86b0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.510394 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a4d6c496214cb78005877727536301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.512516 15609 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Bootstrap starting.
I20260926 12:29:00.512538 15611 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: Bootstrap starting.
I20260926 12:29:00.513355 15611 tablet_bootstrap.cc:654] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.513417 15609 tablet_bootstrap.cc:654] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.514392 15515 tablet_service.cc:1467] Processing CreateTablet for tablet 12a4d6c496214cb78005877727536301 (DEFAULT_TABLE table=test-table [id=f271dae8300747fc95604c94cb86b0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:00.514635 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a4d6c496214cb78005877727536301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:00.514704 15611 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: No bootstrap required, opened a new log
I20260926 12:29:00.514778 15611 ts_tablet_manager.cc:1397] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:29:00.515162 15611 raft_consensus.cc:348] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.515271 15611 raft_consensus.cc:374] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.515304 15611 raft_consensus.cc:729] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.515420 15611 consensus_queue.cc:260] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.515679 15611 ts_tablet_manager.cc:1428] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.516777 15610 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Bootstrap starting.
I20260926 12:29:00.517603 15610 tablet_bootstrap.cc:654] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:00.518918 15609 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: No bootstrap required, opened a new log
I20260926 12:29:00.519001 15609 ts_tablet_manager.cc:1397] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 12:29:00.519429 15609 raft_consensus.cc:348] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.519572 15609 raft_consensus.cc:374] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.519603 15609 raft_consensus.cc:729] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe97be3f2564511bb49824c1c1c151c, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.519719 15609 consensus_queue.cc:260] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.519955 15609 ts_tablet_manager.cc:1428] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.520866 15610 tablet_bootstrap.cc:492] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: No bootstrap required, opened a new log
I20260926 12:29:00.520964 15610 ts_tablet_manager.cc:1397] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 12:29:00.521360 15610 raft_consensus.cc:348] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.521467 15610 raft_consensus.cc:374] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:00.521502 15610 raft_consensus.cc:729] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03698c5d081445ce8e7977ddee069216, State: Initialized, Role: FOLLOWER
I20260926 12:29:00.521613 15610 consensus_queue.cc:260] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.521860 15610 ts_tablet_manager.cc:1428] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:29:00.581197 15620 raft_consensus.cc:482] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:29:00.581357 15620 raft_consensus.cc:504] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.581789 15620 leader_election.cc:288] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339), 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885)
I20260926 12:29:00.582264 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a4d6c496214cb78005877727536301" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c" is_pre_election: true
I20260926 12:29:00.582263 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a4d6c496214cb78005877727536301" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216" is_pre_election: true
I20260926 12:29:00.582423 15535 raft_consensus.cc:2454] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 0.
I20260926 12:29:00.582430 15395 raft_consensus.cc:2454] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 0.
I20260926 12:29:00.582702 15189 leader_election.cc:302] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2af0a3f0f54be3804ecb992d66cfe8, 4fe97be3f2564511bb49824c1c1c151c; no voters:
I20260926 12:29:00.582875 15620 raft_consensus.cc:2781] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:29:00.582942 15620 raft_consensus.cc:482] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:29:00.582978 15620 raft_consensus.cc:3037] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.584255 15620 raft_consensus.cc:504] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.584600 15620 leader_election.cc:288] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339), 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885)
I20260926 12:29:00.584972 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a4d6c496214cb78005877727536301" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe97be3f2564511bb49824c1c1c151c"
I20260926 12:29:00.585083 15395 raft_consensus.cc:3037] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.585057 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a4d6c496214cb78005877727536301" candidate_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03698c5d081445ce8e7977ddee069216"
I20260926 12:29:00.585163 15535 raft_consensus.cc:3037] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:00.586119 15395 raft_consensus.cc:2454] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 1.
I20260926 12:29:00.586208 15535 raft_consensus.cc:2454] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2af0a3f0f54be3804ecb992d66cfe8 in term 1.
I20260926 12:29:00.586396 15189 leader_election.cc:302] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2af0a3f0f54be3804ecb992d66cfe8, 4fe97be3f2564511bb49824c1c1c151c; no voters:
I20260926 12:29:00.586570 15620 raft_consensus.cc:2781] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:00.586727 15620 raft_consensus.cc:686] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2af0a3f0f54be3804ecb992d66cfe8, State: Running, Role: LEADER
I20260926 12:29:00.586871 15620 consensus_queue.cc:237] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } }
I20260926 12:29:00.588598 15098 catalog_manager.cc:5495] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2af0a3f0f54be3804ecb992d66cfe8" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 } health_report { overall_health: UNKNOWN } } }
I20260926 12:29:00.615021 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.611556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.7.99.1:45741)
0926 12:29:00.611678 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.611682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.611746 (+ 64us) server_negotiation.cc:407] Connection header received
0926 12:29:00.611886 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.611888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.611927 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.611989 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.612294 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.612844 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.614027 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.614228 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.614301 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.614576 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.614638 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.614699 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.614831 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.614874 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 12:29:00.625380 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.621955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:54373 (local address 127.7.99.3:32885)
0926 12:29:00.622074 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.622078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.622176 (+ 98us) server_negotiation.cc:407] Connection header received
0926 12:29:00.622309 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.622311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.622352 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.622411 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.622695 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.623173 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.624414 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.624612 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.624686 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.624998 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.625061 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.625111 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.625235 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.625276 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:29:00.625474 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.621799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:32885 (local address 127.7.99.1:54373)
0926 12:29:00.622086 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.622101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.622217 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.622448 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.622451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.622459 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:29:00.622589 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.622595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.623332 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.623336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.624304 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.624316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.624729 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.624733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.624819 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.624872 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.625146 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.625152 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.625231 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.625322 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260926 12:29:00.630846 15395 raft_consensus.cc:1261] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Refusing update from remote peer 3e2af0a3f0f54be3804ecb992d66cfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.630872 15535 raft_consensus.cc:1261] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Refusing update from remote peer 3e2af0a3f0f54be3804ecb992d66cfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:00.631206 15620 consensus_queue.cc:1035] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe97be3f2564511bb49824c1c1c151c" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.631377 15707 consensus_queue.cc:1035] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03698c5d081445ce8e7977ddee069216" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:00.649392 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.645733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45741 (local address 127.7.99.3:58331)
0926 12:29:00.645957 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.645973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:00.646045 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:00.646503 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:00.646507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:00.646527 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:29:00.646636 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.646643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.647372 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.647379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.648056 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:00.648064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.648627 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.648631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:00.648728 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.648764 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.649068 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.649074 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:29:00.649136 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 12:29:00.649216 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 12:29:00.649793 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.646192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:58331 (local address 127.7.99.1:45741)
0926 12:29:00.646287 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:00.646292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:00.646304 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:29:00.646342 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:00.646345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:00.646411 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:00.646494 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:00.646745 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.647273 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.648192 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:00.648504 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:00.648621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:00.648873 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:00.648953 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:00.649028 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:00.649663 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 12:29:00.649705 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:29:00.669371 15098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.669515 15098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:00.671372 15098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4721ca5fb37b4ef5ae70d7f490c6f680: Sending DeleteTablet for 3 replicas of tablet 12a4d6c496214cb78005877727536301
I20260926 12:29:00.671871 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a4d6c496214cb78005877727536301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:56338
I20260926 12:29:00.671967 15515 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a4d6c496214cb78005877727536301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50588
I20260926 12:29:00.671983 15375 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a4d6c496214cb78005877727536301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:00 UTC) from {username='oryx', principal='oryx/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48398
I20260926 12:29:00.672051 15682 tablet_replica.cc:330] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: stopping tablet replica
I20260926 12:29:00.672137 15684 tablet_replica.cc:330] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: stopping tablet replica
I20260926 12:29:00.672187 15681 tablet_replica.cc:330] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: stopping tablet replica
I20260926 12:29:00.672194 15682 raft_consensus.cc:2229] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:29:00.672279 15684 raft_consensus.cc:2229] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.672284 15681 raft_consensus.cc:2229] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:00.672374 15684 raft_consensus.cc:2258] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.672365 15682 raft_consensus.cc:2258] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.672379 15681 raft_consensus.cc:2258] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:00.673017 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15170
I20260926 12:29:00.673156 15682 ts_tablet_manager.cc:1905] T 12a4d6c496214cb78005877727536301 P 3e2af0a3f0f54be3804ecb992d66cfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.673259 15681 ts_tablet_manager.cc:1905] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.675469 15684 ts_tablet_manager.cc:1905] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:00.676435 15681 ts_tablet_manager.cc:1918] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:29:00.676529 15681 log.cc:1196] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/12a4d6c496214cb78005877727536301
I20260926 12:29:00.676796 15681 ts_tablet_manager.cc:1939] T 12a4d6c496214cb78005877727536301 P 4fe97be3f2564511bb49824c1c1c151c: Deleting consensus metadata
I20260926 12:29:00.677587 15684 ts_tablet_manager.cc:1918] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:29:00.677587 15085 catalog_manager.cc:4841] TS 4fe97be3f2564511bb49824c1c1c151c (127.7.99.2:40339): tablet 12a4d6c496214cb78005877727536301 (table test-table [id=f271dae8300747fc95604c94cb86b0d4]) successfully deleted
I20260926 12:29:00.677675 15684 log.cc:1196] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/12a4d6c496214cb78005877727536301
I20260926 12:29:00.677897 15684 ts_tablet_manager.cc:1939] T 12a4d6c496214cb78005877727536301 P 03698c5d081445ce8e7977ddee069216: Deleting consensus metadata
I20260926 12:29:00.678483 15084 catalog_manager.cc:4841] TS 03698c5d081445ce8e7977ddee069216 (127.7.99.3:32885): tablet 12a4d6c496214cb78005877727536301 (table test-table [id=f271dae8300747fc95604c94cb86b0d4]) successfully deleted
W20260926 12:29:00.681557 15082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:45741 (error 108)
I20260926 12:29:00.681650 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15310
I20260926 12:29:00.683905 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:00.683455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45741 (local address 127.0.0.1:56364)
0926 12:29:00.683690 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:00.683777 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:45741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":51,"threads_started":1}
W20260926 12:29:00.684057 15082 catalog_manager.cc:4584] TS 3e2af0a3f0f54be3804ecb992d66cfe8 (127.7.99.1:45741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12a4d6c496214cb78005877727536301: Network error: Client connection negotiation failed: client connection to 127.7.99.1:45741: connect: Connection refused (error 111)
I20260926 12:29:00.688529 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15450
I20260926 12:29:00.695124 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15066
2026-09-26T12:29:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3170 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15720](info): set up 2 sockets
Sep 26 12:29:00 dist-test-slave-3jw2 krb5kdc[15720](info): commencing operation
krb5kdc: starting...
W20260926 12:29:01.729192 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:29:01 dist-test-slave-3jw2 krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:29:01.846848 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38143
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:38143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:01.954593 15733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:01.954826 15733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:01.954862 15733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:01.958722 15733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:01.958765 15733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:01.958783 15733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:01.958798 15733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:01.958812 15733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:01.963634 15733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:38143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:01.964527 15733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:01.965416 15733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:29:01.973869 15733 server_base.cc:1034] running on GCE node
W20260926 12:29:01.973838 15738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:01.973860 15739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:01.973839 15741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:01.974483 15733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:01.975449 15733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:01.976610 15733 hybrid_clock.cc:648] HybridClock initialized: now 1790425741976561 us; error 59 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T12:29:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1296 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:29:02 dist-test-slave-3jw2 krb5kdc[15746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:29:02 dist-test-slave-3jw2 krb5kdc[15746](info): set up 2 sockets
Sep 26 12:29:02 dist-test-slave-3jw2 krb5kdc[15746](info): commencing operation
krb5kdc: starting...
W20260926 12:29:03.045272 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:29:03.165239 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43399
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:03.275969 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:03.276329 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:03.276422 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:03.280325 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:03.280414 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:03.280457 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:03.280521 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:03.280563 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:03.285670 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:03.286770 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:03.287771 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:03.296155 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.296180 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.296365 15759 server_base.cc:1034] running on GCE node
W20260926 12:29:03.296182 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.297065 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:03.298079 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:03.299268 15759 hybrid_clock.cc:648] HybridClock initialized: now 1790425743299258 us; error 27 us; skew 500 ppm
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:03.301858 15759 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:29:03.302657 15759 webserver.cc:460] Webserver started at http://127.7.99.62:39087/ using document root <none> and password file <none>
I20260926 12:29:03.303222 15759 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:03.303310 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:03.303551 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:03.305488 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "dec41799a20b40d584982b06f0734545"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "bac8880a6c72d950da2d284ca894beef"
server_key_iv: "b760ab81e2834091dd42749727cce303"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.306048 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "dec41799a20b40d584982b06f0734545"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "bac8880a6c72d950da2d284ca894beef"
server_key_iv: "b760ab81e2834091dd42749727cce303"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.309517 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:29:03.311756 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.312983 15759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:03.313130 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "dec41799a20b40d584982b06f0734545"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "bac8880a6c72d950da2d284ca894beef"
server_key_iv: "b760ab81e2834091dd42749727cce303"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.313233 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:03.324283 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:03.326924 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:03.327109 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:03.334002 15759 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43399
I20260926 12:29:03.334065 15826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43399 every 8 connection(s)
I20260926 12:29:03.334873 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:29:03.337448 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:03.341676 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15759
I20260926 12:29:03.341804 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:29:03.342087 7564 external_mini_cluster.cc:1428] Setting key 90e2a2204658f37af007026682be94c5
I20260926 12:29:03.343179 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Bootstrap starting.
I20260926 12:29:03.345434 15827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:03.346225 15827 log.cc:824] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:03.348325 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: No bootstrap required, opened a new log
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:03.353121 15827 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:03.353392 15827 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:03.353464 15827 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec41799a20b40d584982b06f0734545, State: Initialized, Role: FOLLOWER
I20260926 12:29:03.353914 15827 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:03.354059 15827 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:29:03.354123 15827 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:29:03.354214 15827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:03.355019 15827 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:03.355370 15827 leader_election.cc:302] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec41799a20b40d584982b06f0734545; no voters:
I20260926 12:29:03.355707 15827 leader_election.cc:288] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:29:03.355788 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:03.356000 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 LEADER]: Becoming Leader. State: Replica: dec41799a20b40d584982b06f0734545, State: Running, Role: LEADER
I20260926 12:29:03.356334 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:03.343399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.7.99.62:43399)
0926 12:29:03.343772 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:03.343795 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:03.343828 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:29:03.344442 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:03.344458 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:03.344759 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:03.345008 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:03.345908 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.346900 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.348008 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.348534 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.348654 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:03.351281 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:03.351290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:03.351320 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:03.351355 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:03.353305 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.353777 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.353787 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.353791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.353900 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.354198 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.354205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.354207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.354621 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:03.354826 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:03.355123 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 12:29:03.355387 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":139,"threads_started":1}
I20260926 12:29:03.356354 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:03.356966 15827 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:29:03.358197 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec41799a20b40d584982b06f0734545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } } }
I20260926 12:29:03.358318 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:03.358704 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec41799a20b40d584982b06f0734545. Latest consensus state: current_term: 1 leader_uuid: "dec41799a20b40d584982b06f0734545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } } }
I20260926 12:29:03.358852 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:03.359153 15841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:29:03.361855 15841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:29:03.366719 15841 catalog_manager.cc:1347] Generated new cluster ID: 8fb113a17bb04311b3d0a51a29a965f8
I20260926 12:29:03.366787 15841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:29:03.383591 15841 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:29:03.384454 15841 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:29:03.391360 15841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Generated new TSK 0
I20260926 12:29:03.391969 15841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:29:03.425311 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:03.532821 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:03.533036 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:03.533082 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:03.536892 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:03.536937 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:03.536990 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:29:03.541934 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:03.542752 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:03.543603 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:03.552569 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.552613 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.552737 15853 server_base.cc:1034] running on GCE node
W20260926 12:29:03.552582 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.553179 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:03.553817 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:03.554996 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790425743554980 us; error 27 us; skew 500 ppm
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:03.557849 15853 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:29:03.558660 15853 webserver.cc:460] Webserver started at http://127.7.99.1:37273/ using document root <none> and password file <none>
I20260926 12:29:03.559194 15853 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:03.559249 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:03.559396 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:03.561256 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "b1a6aec30de04f149473144a09e45a63"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "25189d1d24ba6376ad7bf552efb019c8"
server_key_iv: "e5e6d8006d2e5d6934c498329b888082"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.561724 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a6aec30de04f149473144a09e45a63"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "25189d1d24ba6376ad7bf552efb019c8"
server_key_iv: "e5e6d8006d2e5d6934c498329b888082"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.565255 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:29:03.567368 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.568660 15853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:03.568769 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "b1a6aec30de04f149473144a09e45a63"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "25189d1d24ba6376ad7bf552efb019c8"
server_key_iv: "e5e6d8006d2e5d6934c498329b888082"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.568877 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:03.581451 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:03.584045 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:03.584252 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:03.584815 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:03.585592 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:03.585634 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.585716 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:03.585747 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.594044 15853 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:42893
I20260926 12:29:03.594079 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:42893 every 8 connection(s)
I20260926 12:29:03.594878 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:29:03.600895 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15853
I20260926 12:29:03.600993 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:29:03.601289 7564 external_mini_cluster.cc:1428] Setting key 0f32b7370e90495c8751df78c59a33e2
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:03.608057 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:03.596639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42957 (local address 127.7.99.62:43399)
0926 12:29:03.596766 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:03.596769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:03.597739 (+ 970us) server_negotiation.cc:407] Connection header received
0926 12:29:03.598555 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:03.598558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:03.598600 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:03.598673 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:03.599773 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.600292 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.601505 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.601795 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.601927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:03.604676 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:03.604679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:03.604681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:03.604702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:03.606089 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.606622 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.606630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.606632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.606698 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.607042 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.607045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.607046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.607214 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:03.607323 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:03.607781 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 12:29:03.607896 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:29:03.608877 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:03.596984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:42957)
0926 12:29:03.597625 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:03.597662 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:03.598370 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:03.598823 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:03.598829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:03.599165 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:03.599653 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:03.599666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.600399 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.600402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:03.601366 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:03.601384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.601965 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.601968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:03.602159 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:03.602839 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:03.602855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:03.604505 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:03.606214 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:03.606219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:03.606232 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:03.606531 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:03.606788 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:03.606792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:03.606802 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:03.606939 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:03.607324 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:03.607331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:03.607562 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 12:29:03.607742 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:29:03.608047 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":114,"threads_started":1}
I20260926 12:29:03.609975 15982 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43399
I20260926 12:29:03.610208 15982 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:29:03.610704 15982 heartbeater.cc:511] Master 127.7.99.62:43399 requested a full tablet report, sending...
I20260926 12:29:03.612112 15791 ts_manager.cc:194] Registered new tserver with Master: b1a6aec30de04f149473144a09e45a63 (127.7.99.1:42893)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:29:03.613510 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:42957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:29:03.629011 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:03.733973 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:03.734234 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:03.734310 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:03.738232 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:03.738317 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:03.738410 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:29:03.743319 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:03.744261 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:03.745270 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:03.754508 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.754527 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.754516 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.754603 15987 server_base.cc:1034] running on GCE node
I20260926 12:29:03.755102 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:03.755724 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:03.756899 15987 hybrid_clock.cc:648] HybridClock initialized: now 1790425743756887 us; error 26 us; skew 500 ppm
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:03.760020 15987 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:29:03.760886 15987 webserver.cc:460] Webserver started at http://127.7.99.2:38255/ using document root <none> and password file <none>
I20260926 12:29:03.761370 15987 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:03.761430 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:03.761580 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:03.763465 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "f15357638431497bac8aa49e0e1c9cdc"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "dec586a28488d9d1f7f1feef02690087"
server_key_iv: "515dab8559a3ee39b5a1944d2d27c0d0"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.763994 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "f15357638431497bac8aa49e0e1c9cdc"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "dec586a28488d9d1f7f1feef02690087"
server_key_iv: "515dab8559a3ee39b5a1944d2d27c0d0"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.767359 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:29:03.769658 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.770779 15987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:29:03.770892 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "f15357638431497bac8aa49e0e1c9cdc"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "dec586a28488d9d1f7f1feef02690087"
server_key_iv: "515dab8559a3ee39b5a1944d2d27c0d0"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.770987 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:03.784145 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:03.786762 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:03.786952 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:03.787528 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:03.788347 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:03.788393 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.788461 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:03.788499 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.797168 15987 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:38171
I20260926 12:29:03.797204 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:38171 every 8 connection(s)
I20260926 12:29:03.797971 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:29:03.804565 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 15987
I20260926 12:29:03.804682 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:29:03.804977 7564 external_mini_cluster.cc:1428] Setting key f4efac88aea2f3fbdddbd4c528432aad
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:03.812106 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:03.799602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:45669 (local address 127.7.99.62:43399)
0926 12:29:03.799724 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:03.799728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:03.800990 (+ 1262us) server_negotiation.cc:407] Connection header received
0926 12:29:03.801900 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:03.801904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:03.801957 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:03.802067 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:03.803125 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.803676 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.805156 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.805371 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.805487 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:03.808236 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:03.808243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:03.808247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:03.808280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:03.809884 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.810358 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.810364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.810369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.810462 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:03.810828 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:03.810831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:03.810833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:03.811074 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:03.811182 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:03.811717 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 12:29:03.811855 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:29:03.812706 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:03.799951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:45669)
0926 12:29:03.800857 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:03.800895 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:03.801651 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:03.802228 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:03.802234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:03.802569 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:03.802992 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:03.803003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.803815 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.803817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:03.805015 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:03.805027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:03.805533 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:03.805537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:03.805746 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:03.806379 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:03.806399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:03.808064 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:03.810005 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:03.810010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:03.810020 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:03.810257 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:03.810583 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:03.810588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:03.810591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:03.810731 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:03.811179 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:03.811184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:03.811474 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 12:29:03.811625 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 12:29:03.811880 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":590,"threads_started":1}
I20260926 12:29:03.813843 16116 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43399
I20260926 12:29:03.814065 16116 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:03.814500 16116 heartbeater.cc:511] Master 127.7.99.62:43399 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:29:03.815464 15791 ts_manager.cc:194] Registered new tserver with Master: f15357638431497bac8aa49e0e1c9cdc (127.7.99.2:38171)
I20260926 12:29:03.816020 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:45669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:29:03.831773 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:03.939435 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:03.939703 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:03.939782 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:03.943604 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:03.943684 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:03.943764 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:29:03.948864 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:03.949988 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:03.951164 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:03.960850 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.960850 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:03.960886 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:03.961182 16121 server_base.cc:1034] running on GCE node
I20260926 12:29:03.961560 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:03.962189 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:03.963374 16121 hybrid_clock.cc:648] HybridClock initialized: now 1790425743963335 us; error 50 us; skew 500 ppm
Sep 26 12:29:03 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:03.966089 16121 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:29:03.967023 16121 webserver.cc:460] Webserver started at http://127.7.99.3:46055/ using document root <none> and password file <none>
I20260926 12:29:03.967593 16121 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:03.967679 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:03.967922 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:03.969995 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "36701886c8604312b6bb135a7b870b76"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "3e72c3774e66e19d54fcd677ed4356db"
server_key_iv: "58b7ab613af7e415b32bfd441fc9cda6"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.970546 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "36701886c8604312b6bb135a7b870b76"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "3e72c3774e66e19d54fcd677ed4356db"
server_key_iv: "58b7ab613af7e415b32bfd441fc9cda6"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.974417 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:29:03.976821 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:03.978034 16121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:03.978163 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "36701886c8604312b6bb135a7b870b76"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "3e72c3774e66e19d54fcd677ed4356db"
server_key_iv: "58b7ab613af7e415b32bfd441fc9cda6"
server_key_version: "encryptionkey@0"
I20260926 12:29:03.978271 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:04.006263 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:04.009151 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:04.009366 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:04.009940 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:04.010771 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:04.010848 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.010943 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:04.010995 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.020009 16121 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:42061
I20260926 12:29:04.020035 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:42061 every 8 connection(s)
I20260926 12:29:04.020925 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:29:04.028020 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16121
I20260926 12:29:04.028167 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:29:04.028462 7564 external_mini_cluster.cc:1428] Setting key 1458e95d644ccbb77ed6fc5dc7697cf1
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425743, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:04.034149 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.022628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:33155 (local address 127.7.99.62:43399)
0926 12:29:04.022757 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.022760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.023548 (+ 788us) server_negotiation.cc:407] Connection header received
0926 12:29:04.024796 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.024800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.024856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.024971 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.026053 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.026573 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.027681 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.027899 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.028014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.030685 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:04.030691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:04.030699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:04.030721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:04.032222 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:04.032696 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:04.032700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:04.032702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:04.032779 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:04.033131 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:04.033136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:04.033137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:04.033311 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:04.033424 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:04.033874 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 12:29:04.033986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:29:04.034874 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.022988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:33155)
0926 12:29:04.023395 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.023444 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.024594 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.025127 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.025134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.025471 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.025910 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.025923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.026693 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.026696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.027556 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.027565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.028046 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.028050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.028294 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.028882 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.028898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.030539 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:04.032340 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:04.032345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:04.032358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:04.032600 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:04.032891 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:04.032894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:04.032896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:04.033039 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:04.033413 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:04.033419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:04.033649 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 12:29:04.033823 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 12:29:04.034124 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260926 12:29:04.035931 16250 heartbeater.cc:347] Connected to a master server at 127.7.99.62:43399
I20260926 12:29:04.036230 16250 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:04.036692 16250 heartbeater.cc:511] Master 127.7.99.62:43399 requested a full tablet report, sending...
I20260926 12:29:04.037662 15791 ts_manager.cc:194] Registered new tserver with Master: 36701886c8604312b6bb135a7b870b76 (127.7.99.3:42061)
I20260926 12:29:04.038208 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:33155
I20260926 12:29:04.042104 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
Entry for principal oryx/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab.
WARNING: no policy specified for oryx/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
Entry for principal oryx/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab.
WARNING: no policy specified for oryx/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
Entry for principal oryx/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab.
I20260926 12:29:04.113394 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15853
I20260926 12:29:04.119244 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15987
I20260926 12:29:04.124435 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16121
I20260926 12:29:04.129468 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 15759
I20260926 12:29:04.135484 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43399
--webserver_interface=127.7.99.62
--webserver_port=39087
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:04.243095 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:04.243330 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:04.243366 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:04.247838 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:04.247890 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:04.247910 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:04.247929 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:04.247944 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:04.253130 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:04.254118 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:04.255121 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:04.263448 16265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.263463 16264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.263468 16267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:04.263513 16259 server_base.cc:1034] running on GCE node
I20260926 12:29:04.264268 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:04.265278 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:04.266443 16259 hybrid_clock.cc:648] HybridClock initialized: now 1790425744266425 us; error 29 us; skew 500 ppm
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425744, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:04.269155 16259 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:29:04.270030 16259 webserver.cc:460] Webserver started at http://127.7.99.62:39087/ using document root <none> and password file <none>
I20260926 12:29:04.270583 16259 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:04.270680 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:04.274412 16259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 12:29:04.276401 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.277520 16259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:04.277653 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "dec41799a20b40d584982b06f0734545"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "bac8880a6c72d950da2d284ca894beef"
server_key_iv: "b760ab81e2834091dd42749727cce303"
server_key_version: "encryptionkey@0"
I20260926 12:29:04.278167 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:04.287133 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:04.289917 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:04.290105 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:04.296727 16259 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43399
I20260926 12:29:04.296794 16326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43399 every 8 connection(s)
I20260926 12:29:04.297734 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:29:04.301100 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16259
I20260926 12:29:04.301579 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:42893
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:43399
--webserver_port=37273
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:29:04.302145 16327 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:29:04.304147 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Bootstrap starting.
I20260926 12:29:04.320520 16327 log.cc:824] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:04.327334 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:29:04.327718 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Bootstrap complete.
I20260926 12:29:04.331843 16327 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:04.332286 16327 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec41799a20b40d584982b06f0734545, State: Initialized, Role: FOLLOWER
I20260926 12:29:04.332767 16327 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:04.332880 16327 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:29:04.332966 16327 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:29:04.333061 16327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:29:04.335003 16327 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:04.335322 16327 leader_election.cc:302] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec41799a20b40d584982b06f0734545; no voters:
I20260926 12:29:04.335654 16327 leader_election.cc:288] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:29:04.335760 16331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:29:04.335992 16331 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [term 2 LEADER]: Becoming Leader. State: Replica: dec41799a20b40d584982b06f0734545, State: Running, Role: LEADER
I20260926 12:29:04.336350 16331 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } }
I20260926 12:29:04.336850 16327 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:29:04.337692 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dec41799a20b40d584982b06f0734545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } } }
I20260926 12:29:04.337818 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:04.338097 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec41799a20b40d584982b06f0734545. Latest consensus state: current_term: 2 leader_uuid: "dec41799a20b40d584982b06f0734545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec41799a20b40d584982b06f0734545" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43399 } } }
I20260926 12:29:04.338238 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:04.338514 16340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:29:04.341622 16340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:29:04.342480 16340 catalog_manager.cc:1259] Loaded cluster ID: 8fb113a17bb04311b3d0a51a29a965f8
I20260926 12:29:04.342538 16340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:29:04.344787 16340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:29:04.345420 16340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dec41799a20b40d584982b06f0734545: Loaded TSK: 0
I20260926 12:29:04.346714 16340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:29:04.414515 16329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:04.414786 16329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:04.414899 16329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:04.418982 16329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:04.419034 16329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:04.419122 16329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:29:04.424285 16329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.1.keytab
--principal=oryx/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:42893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=37273
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:04.425230 16329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:04.426201 16329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:04.436576 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:04.436650 16329 server_base.cc:1034] running on GCE node
W20260926 12:29:04.436769 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.436594 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:04.437318 16329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:04.437973 16329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:04.439160 16329 hybrid_clock.cc:648] HybridClock initialized: now 1790425744439132 us; error 44 us; skew 500 ppm
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425744, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:04.442052 16329 init.cc:377] Logged in from keytab as oryx/127.7.99.1@KRBTEST.COM (short username oryx)
I20260926 12:29:04.442852 16329 webserver.cc:460] Webserver started at http://127.7.99.1:37273/ using document root <none> and password file <none>
I20260926 12:29:04.443404 16329 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:04.443512 16329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:04.447363 16329 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:04.449501 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.450736 16329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:29:04.450870 16329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "b1a6aec30de04f149473144a09e45a63"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "25189d1d24ba6376ad7bf552efb019c8"
server_key_iv: "e5e6d8006d2e5d6934c498329b888082"
server_key_version: "encryptionkey@0"
I20260926 12:29:04.451392 16329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:04.462430 16329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:04.465507 16329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:04.465687 16329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:04.466208 16329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:04.466974 16329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:04.467062 16329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.467130 16329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:04.467166 16329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.476605 16329 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:42893
I20260926 12:29:04.476624 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:42893 every 8 connection(s)
I20260926 12:29:04.477603 16329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:29:04.478015 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16329
I20260926 12:29:04.478462 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:38171
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:43399
--webserver_port=38255
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:29:04.492524 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.479581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:46137 (local address 127.7.99.62:43399)
0926 12:29:04.481720 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.481738 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.481782 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:29:04.482191 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.482208 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.482510 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.482772 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.484397 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.485364 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.487199 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.487633 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.487741 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.491830 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:46137: BlockingRecv error: recv got EOF from 127.7.99.1:46137 (error 108)
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":125,"threads_started":1}
I20260926 12:29:04.492561 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.479797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:46137)
0926 12:29:04.480348 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.480399 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.481749 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.482924 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.482935 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.483468 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.484159 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.484227 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.485498 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.485504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.487053 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.487066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.487793 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.487798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.488039 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.488844 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.488870 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.491643 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":107,"threads_started":1}
W20260926 12:29:04.492961 16477 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43399 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.99.62:43399: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:04.498560 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.493374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:38737)
0926 12:29:04.493503 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.493516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.493586 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.493828 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.493831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.493981 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.494077 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.494083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.494713 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.494716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.495594 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.495603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.495997 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.496001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.496092 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.496547 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.496560 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.498420 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 12:29:04.498745 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.493494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:38737 (local address 127.7.99.62:43399)
0926 12:29:04.493628 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.493634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.493650 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:29:04.493690 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.493692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.493726 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.493787 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.494173 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.494624 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.495698 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.495888 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.495972 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.498542 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:38737: BlockingRecv error: recv got EOF from 127.7.99.1:38737 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:04.503883 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.499016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:55305)
0926 12:29:04.499354 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.499365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.499463 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.499707 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.499709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.499853 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.499924 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.499929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.500686 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.500693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.501201 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.501208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.501645 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.501648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.501731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.502185 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.502196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.503793 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284}
W20260926 12:29:04.504040 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.499118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:55305 (local address 127.7.99.62:43399)
0926 12:29:04.499234 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.499241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.499435 (+ 194us) server_negotiation.cc:407] Connection header received
0926 12:29:04.499561 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.499565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.499612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.499694 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.500030 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.500587 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.501313 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.501539 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.501644 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.503925 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:55305: BlockingRecv error: recv got EOF from 127.7.99.1:55305 (error 108)
Metrics: {"server-negotiator.queue_time_us":47,"thread_start_us":128,"threads_started":1}
W20260926 12:29:04.504118 16477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:29:04.590844 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:04.591105 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:04.591151 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:04.595492 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:04.595554 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:04.595613 16479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:29:04.600713 16479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.2.keytab
--principal=oryx/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:38171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=38255
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:04.601627 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:04.602567 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:29:04.611819 16479 server_base.cc:1034] running on GCE node
W20260926 12:29:04.611723 16490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.611757 16488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.611723 16487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:04.612403 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:04.613008 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:04.614164 16479 hybrid_clock.cc:648] HybridClock initialized: now 1790425744614152 us; error 24 us; skew 500 ppm
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425744, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:04.616899 16479 init.cc:377] Logged in from keytab as oryx/127.7.99.2@KRBTEST.COM (short username oryx)
I20260926 12:29:04.617710 16479 webserver.cc:460] Webserver started at http://127.7.99.2:38255/ using document root <none> and password file <none>
I20260926 12:29:04.618242 16479 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:04.618372 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:04.621982 16479 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 12:29:04.623881 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.625083 16479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:29:04.625211 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "f15357638431497bac8aa49e0e1c9cdc"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "dec586a28488d9d1f7f1feef02690087"
server_key_iv: "515dab8559a3ee39b5a1944d2d27c0d0"
server_key_version: "encryptionkey@0"
I20260926 12:29:04.625718 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:04.644704 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:04.647625 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:04.647823 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:04.648396 16479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:04.649183 16479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:04.649250 16479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.649312 16479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:04.649366 16479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.658105 16479 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:38171
I20260926 12:29:04.658131 16610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:38171 every 8 connection(s)
I20260926 12:29:04.659027 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:29:04.666451 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16479
I20260926 12:29:04.666944 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:42061
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:43399
--webserver_port=46055
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:29:04.668963 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.660686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:44157 (local address 127.7.99.62:43399)
0926 12:29:04.660820 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.660824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.661712 (+ 888us) server_negotiation.cc:407] Connection header received
0926 12:29:04.662488 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.662490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.662538 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.662606 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.663651 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.664220 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.665248 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.665457 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.665527 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.668847 (+ 3320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:44157: BlockingRecv error: recv got EOF from 127.7.99.2:44157 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:29:04.669538 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.660994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:44157)
0926 12:29:04.661580 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.661616 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.662320 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.662760 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.662767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.663095 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.663526 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.663539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.664335 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.664338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.665151 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.665158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.665571 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.665573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.665704 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.666154 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.666193 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.668735 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":107,"threads_started":1}
W20260926 12:29:04.670027 16611 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43399 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.99.62:43399: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:04.675179 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.670415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:41899)
0926 12:29:04.670561 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.670597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.670669 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.670893 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.670896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.671046 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.671123 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.671128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.671827 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.671831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.672415 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.672423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.672829 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.672832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.672942 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.673409 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.673420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.675050 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 12:29:04.675261 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.670541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:41899 (local address 127.7.99.62:43399)
0926 12:29:04.670685 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.670692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.670707 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:29:04.670746 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.670750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.670790 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.670880 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.671230 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.671708 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.672523 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.672725 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.672828 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.675151 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:41899: BlockingRecv error: recv got EOF from 127.7.99.2:41899 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:04.680362 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.675742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:47401)
0926 12:29:04.675849 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.675862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.675950 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.676191 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.676193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.676328 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.676385 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.676390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.677090 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.677093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.677591 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.677598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.678032 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.678035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.678116 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.678573 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.678583 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.680245 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:29:04.680464 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.675862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:47401 (local address 127.7.99.62:43399)
0926 12:29:04.675983 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.675989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.676003 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:29:04.676034 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.676036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.676069 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.676155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.676488 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.676974 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.677702 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.677929 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.678032 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.680354 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:47401: BlockingRecv error: recv got EOF from 127.7.99.2:47401 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:29:04.680732 16611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:29:04.777168 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:04.777419 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:04.777508 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:04.781469 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:04.781517 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:04.781620 16614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:29:04.786680 16614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/krb5kdc/oryx_127.7.99.3.keytab
--principal=oryx/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:42061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=46055
--tserver_master_addrs=127.7.99.62:43399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:04.787665 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:04.788805 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:04.797760 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.797768 16620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:04.797775 16622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:04.797924 16614 server_base.cc:1034] running on GCE node
I20260926 12:29:04.798344 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:04.798964 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:04.800100 16614 hybrid_clock.cc:648] HybridClock initialized: now 1790425744800065 us; error 54 us; skew 500 ppm
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425744, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:04.802811 16614 init.cc:377] Logged in from keytab as oryx/127.7.99.3@KRBTEST.COM (short username oryx)
I20260926 12:29:04.803638 16614 webserver.cc:460] Webserver started at http://127.7.99.3:46055/ using document root <none> and password file <none>
I20260926 12:29:04.804214 16614 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:04.804283 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:04.807893 16614 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:04.809916 16629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.811075 16614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 12:29:04.811180 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "36701886c8604312b6bb135a7b870b76"
format_stamp: "Formatted at 2026-09-26 12:29:03 on dist-test-slave-3jw2"
server_key: "3e72c3774e66e19d54fcd677ed4356db"
server_key_iv: "58b7ab613af7e415b32bfd441fc9cda6"
server_key_version: "encryptionkey@0"
I20260926 12:29:04.811671 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:04.831480 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:04.834513 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:04.834698 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:04.835258 16614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:04.836078 16614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:04.836124 16614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.836207 16614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:04.836241 16614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:04.845613 16614 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:42061
I20260926 12:29:04.845636 16742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:42061 every 8 connection(s)
I20260926 12:29:04.846580 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:29:04.853595 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16614
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:29:04.857172 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.848310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:60369 (local address 127.7.99.62:43399)
0926 12:29:04.848437 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.848440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.849447 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 12:29:04.850253 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.850256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.850305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.850376 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.851451 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.851950 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.853085 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.853325 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.853431 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.857049 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:60369: BlockingRecv error: recv got EOF from 127.7.99.3:60369 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:29:04.857894 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.848702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:60369)
0926 12:29:04.849308 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.849347 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.850082 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.850540 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.850545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.850875 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.851332 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.851346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.852052 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.852054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.852984 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.852993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.853457 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.853460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.853590 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.854189 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.854209 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.856954 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":103,"threads_started":1}
W20260926 12:29:04.858421 16743 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:43399 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.99.62:43399: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:29:04.864429 16480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.858982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:33047 (local address 127.7.99.62:43399)
0926 12:29:04.859124 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.859129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.859142 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:04.859170 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.859172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.859216 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.859297 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.859702 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.860269 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.861375 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.861579 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.861665 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.864317 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:33047: BlockingRecv error: recv got EOF from 127.7.99.3:33047 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:29:04.864518 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.854641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.7.99.62:43399)
0926 12:29:04.854767 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.854779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.854804 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:29:04.854843 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.854850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.854885 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.854988 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.855847 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.856686 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.857595 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.857901 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.858015 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.859664 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:04.859672 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:04.859688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:04.859715 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:04.861799 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:04.862336 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:04.862343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:04.862349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:04.862500 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:04.863138 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:04.863142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:04.863144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:04.863546 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:04.863752 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:04.864115 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 12:29:04.864356 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:29:04.864683 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.858861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:33047)
0926 12:29:04.858958 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.858971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.859063 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.859323 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.859326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.859471 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.859573 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.859579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.860377 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.860381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.861253 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.861263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.861693 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.861698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.861796 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.862236 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.862246 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.864580 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:04 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:04.870404 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:04.865670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:48169)
0926 12:29:04.865774 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:04.865787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:04.865873 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:04.866243 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:04.866246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:04.866423 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:04.866487 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.866492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.867352 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.867356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.867848 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:04.867856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.868394 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.868398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:04.868479 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.868819 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:04.868829 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:04.870324 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 12:29:04.870603 16482 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:04.865738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:48169 (local address 127.7.99.62:43399)
0926 12:29:04.865953 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:04.865957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:04.865968 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:29:04.865995 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:04.865997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:04.866035 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:04.866121 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:04.866600 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.867234 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.867959 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:04.868244 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:04.868394 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:04.870463 (+ 2069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:48169: BlockingRecv error: recv got EOF from 127.7.99.3:48169 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
W20260926 12:29:04.870617 16743 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:05.510247 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:05.504746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:58501)
0926 12:29:05.505025 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:05.505039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:05.505127 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:05.505364 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:05.505366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:05.505545 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:05.505662 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.505667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.506401 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.506403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.506973 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.506980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.507381 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.507384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.507457 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.508065 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:05.508097 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:05.510130 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":64,"threads_started":1}
W20260926 12:29:05.510391 16748 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:05.504805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:58501 (local address 127.7.99.62:43399)
0926 12:29:05.505063 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:05.505066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:05.505082 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:29:05.505211 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:05.505214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:05.505254 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:05.505328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:05.505756 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.506298 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.507071 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.507289 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.507363 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.510259 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:58501: BlockingRecv error: recv got EOF from 127.7.99.1:58501 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:05.686924 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:05.681390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:37789)
0926 12:29:05.681816 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:05.681828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:05.681957 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:05.682250 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:05.682253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:05.682475 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:05.682564 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.682570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.683292 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.683295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.683884 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.683891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.684396 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.684400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.684493 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.684935 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:05.684945 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:05.686713 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":79,"threads_started":1}
W20260926 12:29:05.686934 16748 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:05.681462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:37789 (local address 127.7.99.62:43399)
0926 12:29:05.681725 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:05.681731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:05.681902 (+ 171us) server_negotiation.cc:407] Connection header received
0926 12:29:05.682073 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:05.682077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:05.682142 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:05.682230 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:05.682672 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.683190 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.683997 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.684298 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.684394 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.686837 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:37789: BlockingRecv error: recv got EOF from 127.7.99.2:37789 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:05 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:05.876773 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:05.871239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:34081)
0926 12:29:05.871496 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:05.871511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:05.871627 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:05.871929 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:05.871932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:05.872137 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:05.872262 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.872269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.873062 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.873067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.873648 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:05.873656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.874165 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.874168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:05.874285 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.874765 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:05.874776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:05.876650 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
W20260926 12:29:05.876895 16748 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:05.871304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:34081 (local address 127.7.99.62:43399)
0926 12:29:05.871458 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:05.871464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:05.871602 (+ 138us) server_negotiation.cc:407] Connection header received
0926 12:29:05.871729 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:05.871734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:05.871780 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:05.871905 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:05.872388 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.872953 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.873772 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:05.874007 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:05.874162 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:05.876781 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:34081: BlockingRecv error: recv got EOF from 127.7.99.3:34081 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:06.519927 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:06.510961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:41541)
0926 12:29:06.511222 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:06.511243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:06.511336 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:06.511587 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:06.511589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:06.511760 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:06.511865 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.511870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.512582 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.512585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.513160 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.513168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.513561 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.513563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.513624 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.517904 (+ 4280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:06.517916 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:06.519741 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":65,"threads_started":1}
W20260926 12:29:06.520001 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:06.511079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:41541 (local address 127.7.99.62:43399)
0926 12:29:06.511309 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:06.511312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:06.511325 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:06.511421 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:06.511423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:06.511469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:06.511541 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:06.511960 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.512494 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.513254 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.513466 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.513537 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.519863 (+ 6326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:41541: BlockingRecv error: recv got EOF from 127.7.99.1:41541 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:06.693481 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:06.687664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:52235)
0926 12:29:06.687961 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:06.687977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:06.688088 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:06.688376 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:06.688380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:06.688589 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:06.688677 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.688683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.689484 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.689488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.690235 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.690244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.690823 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.690827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.690926 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.691391 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:06.691402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:06.693286 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
W20260926 12:29:06.693522 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:06.687774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:52235 (local address 127.7.99.62:43399)
0926 12:29:06.687963 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:06.687968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:06.688033 (+ 65us) server_negotiation.cc:407] Connection header received
0926 12:29:06.688218 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:06.688222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:06.688272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:06.688356 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:06.688786 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.689347 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.690346 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.690685 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.690785 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.693420 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:52235: BlockingRecv error: recv got EOF from 127.7.99.2:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:06 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:06.883193 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:06.877618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:39827)
0926 12:29:06.877901 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:06.877920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:06.878034 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:06.878305 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:06.878307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:06.878490 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:06.878587 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.878593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.879309 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.879312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.879950 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:06.879958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.880542 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.880545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:06.880609 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.881147 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:06.881159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:06.882999 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
W20260926 12:29:06.883210 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:06.877800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:39827 (local address 127.7.99.62:43399)
0926 12:29:06.877942 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:06.877946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:06.877988 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:29:06.878119 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:06.878121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:06.878182 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:06.878244 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:06.878690 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.879197 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.880066 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:06.880295 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:06.880371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:06.883116 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:39827: BlockingRecv error: recv got EOF from 127.7.99.3:39827 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:07.530324 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:07.520805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:48425)
0926 12:29:07.521096 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:07.521116 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:07.521212 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:07.521448 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:07.521450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:07.521625 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:07.521727 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.521738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.522509 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.522511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.523120 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.523130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.523544 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.523546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.523605 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.527981 (+ 4376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:07.528020 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:07.530203 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
W20260926 12:29:07.530558 16756 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:07.520885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:48425 (local address 127.7.99.62:43399)
0926 12:29:07.521108 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:07.521112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:07.521185 (+ 73us) server_negotiation.cc:407] Connection header received
0926 12:29:07.521291 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:07.521292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:07.521334 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:07.521416 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:07.521869 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.522405 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.523228 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.523450 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.523525 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.530400 (+ 6875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:48425: BlockingRecv error: recv got EOF from 127.7.99.1:48425 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:07.700717 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:07.694288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:39455)
0926 12:29:07.694618 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:07.694635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:07.694785 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:07.695089 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:07.695092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:07.695299 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:07.695436 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.695443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.696212 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.696215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.697177 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.697187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.697776 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.697779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.697870 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.698344 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:07.698356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:07.700576 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":80,"threads_started":1}
W20260926 12:29:07.700819 16756 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:07.694400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:39455 (local address 127.7.99.62:43399)
0926 12:29:07.694596 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:07.694601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:07.694725 (+ 124us) server_negotiation.cc:407] Connection header received
0926 12:29:07.694919 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:07.694924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:07.694979 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:07.695069 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:07.695548 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.696080 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.697330 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.697661 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.697774 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.700699 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:39455: BlockingRecv error: recv got EOF from 127.7.99.2:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:07 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:07.889007 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:07.883973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:35215)
0926 12:29:07.884247 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:07.884260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:07.884357 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:07.884586 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:07.884588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:07.884761 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:07.884860 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.884865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.885552 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.885554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.886119 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:07.886126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.886513 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.886515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:07.886574 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.887029 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:07.887040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:07.888899 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
W20260926 12:29:07.889135 16756 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:07.884064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35215 (local address 127.7.99.62:43399)
0926 12:29:07.884238 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:07.884242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:07.884329 (+ 87us) server_negotiation.cc:407] Connection header received
0926 12:29:07.884442 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:07.884444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:07.884487 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:07.884553 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:07.884957 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.885463 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.886211 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:07.886424 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:07.886497 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:07.889022 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:35215: BlockingRecv error: recv got EOF from 127.7.99.3:35215 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:08.536765 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:08.531160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:45541)
0926 12:29:08.531465 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:08.531482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:08.531612 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:08.531928 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:08.531931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:08.532207 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:08.532336 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.532344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.533235 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.533238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.533835 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.533844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.534261 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.534264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.534326 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.534812 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:08.534823 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:08.536650 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
W20260926 12:29:08.536868 16760 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:08.531250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:45541 (local address 127.7.99.62:43399)
0926 12:29:08.531497 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:08.531501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:08.531575 (+ 74us) server_negotiation.cc:407] Connection header received
0926 12:29:08.531717 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:08.531720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:08.531785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:08.531859 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:08.532444 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.533139 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.533946 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.534168 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.534242 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.536767 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:45541: BlockingRecv error: recv got EOF from 127.7.99.1:45541 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:08.707048 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:08.701485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:49643)
0926 12:29:08.701743 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:08.701757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:08.701851 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:08.702079 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:08.702080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:08.702254 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:08.702347 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.702351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.703097 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.703100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.703690 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.703698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.704119 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.704121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.704268 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.704765 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:08.704780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:08.706884 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20260926 12:29:08.707213 16760 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:08.701570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:49643 (local address 127.7.99.62:43399)
0926 12:29:08.701757 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:08.701760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:08.701840 (+ 80us) server_negotiation.cc:407] Connection header received
0926 12:29:08.701932 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:08.701934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:08.701978 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:08.702044 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:08.702443 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.702978 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.703798 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.704019 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.704098 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.707012 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:49643: BlockingRecv error: recv got EOF from 127.7.99.2:49643 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:08 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:08.895555 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:08.889958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:43623)
0926 12:29:08.890348 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:08.890364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:08.890490 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:08.890815 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:08.890818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:08.891076 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:08.891204 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.891213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.892037 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.892040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.892648 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:08.892656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.893114 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.893117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:08.893179 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.893674 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:08.893685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:08.895440 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":95,"threads_started":1}
W20260926 12:29:08.895689 16760 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:08.890055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:43623 (local address 127.7.99.62:43399)
0926 12:29:08.890212 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:08.890217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:08.890454 (+ 237us) server_negotiation.cc:407] Connection header received
0926 12:29:08.890603 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:08.890605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:08.890679 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:08.890771 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:08.891326 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.891937 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.892750 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:08.892971 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:08.893046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:08.895564 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:43623: BlockingRecv error: recv got EOF from 127.7.99.3:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:09.543233 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:09.537586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:54027)
0926 12:29:09.537851 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:09.537864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:09.538023 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:09.538346 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:09.538348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:09.538520 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:09.538615 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.538620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.539540 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.539545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.540099 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.540107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.540598 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.540602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.540692 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.541148 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:09.541159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:09.543035 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
W20260926 12:29:09.543313 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:09.537634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:54027 (local address 127.7.99.62:43399)
0926 12:29:09.537868 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:09.537872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:09.537974 (+ 102us) server_negotiation.cc:407] Connection header received
0926 12:29:09.538160 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:09.538162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:09.538217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:09.538308 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:09.538738 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.539445 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.540252 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.540491 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.540596 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.543174 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:54027: BlockingRecv error: recv got EOF from 127.7.99.1:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":59,"threads_started":1}
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:09.713677 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:09.707829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:42659)
0926 12:29:09.708142 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:09.708185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:09.708313 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:09.708583 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:09.708586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:09.708801 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:09.708925 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.708932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.709827 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.709831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.710692 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.710703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.711142 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.711145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.711207 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.711675 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:09.711686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:09.713569 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
W20260926 12:29:09.713835 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:09.707998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:42659 (local address 127.7.99.62:43399)
0926 12:29:09.708145 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:09.708151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:09.708243 (+ 92us) server_negotiation.cc:407] Connection header received
0926 12:29:09.708417 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:09.708421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:09.708475 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:09.708561 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:09.709042 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.709708 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.710806 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.711029 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.711107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.713688 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:42659: BlockingRecv error: recv got EOF from 127.7.99.2:42659 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:09 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:09.901834 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:09.896552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:34011)
0926 12:29:09.896936 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:09.896951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:09.897051 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:09.897334 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:09.897337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:09.897511 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:09.897623 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.897629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.898362 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.898364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.898964 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:09.898973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.899378 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.899381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:09.899448 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.899915 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:09.899925 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:09.901680 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":210,"threads_started":1}
W20260926 12:29:09.901901 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:09.896646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:34011 (local address 127.7.99.62:43399)
0926 12:29:09.896785 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:09.896789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:09.897024 (+ 235us) server_negotiation.cc:407] Connection header received
0926 12:29:09.897133 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:09.897135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:09.897179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:09.897249 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:09.897729 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.898237 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.899070 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:09.899280 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:09.899352 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:09.901809 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:34011: BlockingRecv error: recv got EOF from 127.7.99.3:34011 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:10.549343 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:10.544208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:58821)
0926 12:29:10.544463 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:10.544477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:10.544570 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:10.544799 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:10.544801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:10.544975 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:10.545092 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.545097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.545793 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.545796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.546368 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.546374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.546783 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.546786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.546843 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.547384 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:10.547396 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:10.549233 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
W20260926 12:29:10.549501 16768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:10.544269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:58821 (local address 127.7.99.62:43399)
0926 12:29:10.544492 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:10.544495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:10.544524 (+ 29us) server_negotiation.cc:407] Connection header received
0926 12:29:10.544645 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:10.544647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:10.544688 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:10.544764 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:10.545183 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.545706 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.546465 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.546690 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.546768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.549372 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:58821: BlockingRecv error: recv got EOF from 127.7.99.1:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:10.720894 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:10.715052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:58685)
0926 12:29:10.715332 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:10.715347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:10.715467 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:10.715791 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:10.715793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:10.716080 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:10.716305 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.716312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.717207 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.717210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.717758 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.717765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.718191 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.718194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.718248 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.718767 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:10.718787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:10.720779 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
W20260926 12:29:10.721015 16768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:10.714475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:58685 (local address 127.7.99.62:43399)
0926 12:29:10.714678 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:10.714683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:10.715431 (+ 748us) server_negotiation.cc:407] Connection header received
0926 12:29:10.715567 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:10.715570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:10.715650 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:10.715771 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:10.716413 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.717106 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.717873 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.718090 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.718174 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.720904 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:58685: BlockingRecv error: recv got EOF from 127.7.99.2:58685 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:10 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:10.908542 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:10.902653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:48085)
0926 12:29:10.902986 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:10.903020 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:10.903117 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:10.903396 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:10.903400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:10.903615 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:10.903712 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.903718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.904597 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.904604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.905302 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:10.905312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.905812 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.905814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:10.905923 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.906548 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:10.906559 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:10.908425 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":132,"threads_started":1}
W20260926 12:29:10.908663 16768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:10.902860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:48085 (local address 127.7.99.62:43399)
0926 12:29:10.903033 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:10.903037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:10.903075 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:29:10.903212 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:10.903214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:10.903265 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:10.903371 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:10.903855 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.904484 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.905439 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:10.905670 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:10.905803 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:10.908553 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:48085: BlockingRecv error: recv got EOF from 127.7.99.3:48085 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:11.555276 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:11.550082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:33911)
0926 12:29:11.550383 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:11.550404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:11.550515 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:11.550727 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:11.550729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:11.550900 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:11.550994 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.550999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.551704 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.551706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.552334 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.552343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.552780 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.552783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.552841 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.553289 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:11.553300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:11.555096 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
W20260926 12:29:11.555339 16772 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:11.550137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:33911 (local address 127.7.99.62:43399)
0926 12:29:11.550407 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:11.550410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:11.550472 (+ 62us) server_negotiation.cc:407] Connection header received
0926 12:29:11.550593 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:11.550595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:11.550637 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:11.550716 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:11.551083 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.551613 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.552454 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.552681 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.552767 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.555209 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:33911: BlockingRecv error: recv got EOF from 127.7.99.1:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:11.726495 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:11.721640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:47021)
0926 12:29:11.721947 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:11.721960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:11.722057 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:11.722258 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:11.722260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:11.722440 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:11.722527 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.722531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.723205 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.723207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.723720 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.723726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.724117 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.724119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.724236 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.724727 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:11.724738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:11.726390 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":61,"threads_started":1}
W20260926 12:29:11.726619 16772 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:11.721822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:47021 (local address 127.7.99.62:43399)
0926 12:29:11.721936 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:11.721940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:11.722011 (+ 71us) server_negotiation.cc:407] Connection header received
0926 12:29:11.722122 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:11.722124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:11.722164 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:11.722228 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:11.722618 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.723121 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.723815 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.724023 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.724102 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.726509 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:47021: BlockingRecv error: recv got EOF from 127.7.99.2:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:11 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:11.915030 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:11.909503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:60435)
0926 12:29:11.909831 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:11.909843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:11.909978 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:11.910247 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:11.910250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:11.910454 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:11.910563 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.910569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.911350 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.911356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.911920 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:11.911928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.912463 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.912467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:11.912554 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.913029 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:11.913040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:11.914919 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
W20260926 12:29:11.915145 16772 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:11.909568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:60435 (local address 127.7.99.62:43399)
0926 12:29:11.909737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:11.909741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:11.909922 (+ 181us) server_negotiation.cc:407] Connection header received
0926 12:29:11.910078 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:11.910082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:11.910138 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:11.910227 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:11.910683 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.911226 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.912034 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:11.912343 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:11.912448 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:11.915033 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:60435: BlockingRecv error: recv got EOF from 127.7.99.3:60435 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:12.568286 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:12.556043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:40897)
0926 12:29:12.556308 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:12.556321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:12.556407 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:12.556671 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:12.556673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:12.556841 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:12.556944 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.556948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.557715 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.557717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.558280 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.558286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.558704 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.558707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.558762 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.566132 (+ 7370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:12.566143 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:12.568072 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260926 12:29:12.568388 16776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:12.556224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:40897 (local address 127.7.99.62:43399)
0926 12:29:12.556461 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:12.556464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:12.556477 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:12.556513 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:12.556515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:12.556558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:12.556643 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:12.557052 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.557623 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.558384 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.558608 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.558693 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.568252 (+ 9559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:40897: BlockingRecv error: recv got EOF from 127.7.99.1:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:12.733537 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:12.727322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:54211)
0926 12:29:12.727614 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:12.727627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:12.727779 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:12.728067 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:12.728070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:12.728307 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:12.728407 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.728412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.729489 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.729495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.730203 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.730213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.730709 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.730713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.730819 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.731334 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:12.731347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:12.733353 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
W20260926 12:29:12.733599 16776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:12.727377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:54211 (local address 127.7.99.62:43399)
0926 12:29:12.727550 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:12.727555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:12.727707 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:29:12.727893 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:12.727896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:12.727953 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:12.728046 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:12.728526 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.729378 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.730329 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.730588 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.730710 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.733481 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:54211: BlockingRecv error: recv got EOF from 127.7.99.2:54211 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:12 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:12.921624 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:12.915818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:60787)
0926 12:29:12.916090 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:12.916111 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:12.916269 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:12.916543 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:12.916546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:12.916745 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:12.916833 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.916839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.917592 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.917597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.918162 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:12.918170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.918618 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.918621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:12.918704 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.919209 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:12.919243 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:12.921511 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
W20260926 12:29:12.921794 16776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:12.915881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:60787 (local address 127.7.99.62:43399)
0926 12:29:12.916049 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:12.916054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:12.916194 (+ 140us) server_negotiation.cc:407] Connection header received
0926 12:29:12.916368 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:12.916371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:12.916434 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:12.916521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:12.916954 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.917483 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.918284 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:12.918510 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:12.918617 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:12.921641 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:60787: BlockingRecv error: recv got EOF from 127.7.99.3:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:13.574628 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:13.569069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:44237)
0926 12:29:13.569361 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:13.569375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:13.569472 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:13.569742 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:13.569744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:13.569927 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:13.570022 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.570027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.570749 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.570751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.571286 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.571292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.571745 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.571747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.571805 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.572331 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:13.572345 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:13.574517 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
W20260926 12:29:13.574776 16780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:13.569172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:44237 (local address 127.7.99.62:43399)
0926 12:29:13.569422 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:13.569426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:13.569439 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:13.569559 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:13.569561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:13.569628 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:13.569706 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:13.570118 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.570658 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.571406 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.571638 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.571727 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.574648 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:44237: BlockingRecv error: recv got EOF from 127.7.99.1:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":60,"threads_started":1}
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:13.740286 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:13.734337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:39569)
0926 12:29:13.734629 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:13.734644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:13.734798 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:13.735053 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:13.735056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:13.735274 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:13.735390 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.735397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.736142 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.736145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.736856 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.736866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.737329 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.737333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.737424 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.737971 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:13.737984 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:13.740083 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
W20260926 12:29:13.740355 16780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:13.734435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:39569 (local address 127.7.99.62:43399)
0926 12:29:13.734571 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:13.734575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:13.734717 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:29:13.734885 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:13.734895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:13.734937 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:13.735003 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:13.735498 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.736020 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.736963 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.737191 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.737316 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.740249 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:39569: BlockingRecv error: recv got EOF from 127.7.99.2:39569 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:13 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:13.927847 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:13.922374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:45621)
0926 12:29:13.922696 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:13.922710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:13.922838 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:13.923105 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:13.923108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:13.923309 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:13.923407 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.923413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.924134 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.924140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.924727 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:13.924735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.925207 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.925211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:13.925291 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.925827 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:13.925839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:13.927679 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20260926 12:29:13.927901 16780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:13.922480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:45621 (local address 127.7.99.62:43399)
0926 12:29:13.922645 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:13.922650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:13.922780 (+ 130us) server_negotiation.cc:407] Connection header received
0926 12:29:13.922942 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:13.922945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:13.922998 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:13.923083 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:13.923518 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.924031 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.924841 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:13.925065 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:13.925194 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:13.927791 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:45621: BlockingRecv error: recv got EOF from 127.7.99.3:45621 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:14.580847 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:14.575538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:39291)
0926 12:29:14.575806 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:14.575819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:14.575896 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:14.576141 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:14.576143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:14.576359 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:14.576468 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.576473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.577292 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.577294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.577845 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.577852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.578287 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.578290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.578346 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.578859 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:14.578871 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:14.580646 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
W20260926 12:29:14.581013 16784 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:14.575569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:39291 (local address 127.7.99.62:43399)
0926 12:29:14.575861 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:14.575865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:14.575879 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:29:14.575983 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:14.575985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:14.576029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:14.576110 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:14.576569 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.577201 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.577947 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.578194 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.578274 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.580873 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:39291: BlockingRecv error: recv got EOF from 127.7.99.1:39291 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:14.747022 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:14.741162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:34787)
0926 12:29:14.741440 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:14.741454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:14.741579 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:14.741901 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:14.741904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:14.742121 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:14.742234 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.742240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.743061 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.743067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.743677 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.743685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.744205 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.744209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.744311 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.744902 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:14.744917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:14.746913 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260926 12:29:14.747221 16784 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:14.741210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:34787 (local address 127.7.99.62:43399)
0926 12:29:14.741375 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:14.741381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:14.741530 (+ 149us) server_negotiation.cc:407] Connection header received
0926 12:29:14.741692 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:14.741696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:14.741750 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:14.741878 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:14.742366 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.742947 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.743827 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.744056 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.744172 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.747044 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:34787: BlockingRecv error: recv got EOF from 127.7.99.2:34787 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:14 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:14.934072 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:14.928720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:52003)
0926 12:29:14.928979 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:14.928993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:14.929070 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:14.929376 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:14.929378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:14.929604 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:14.929749 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.929755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.930539 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.930541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.931081 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:14.931088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.931533 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.931535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:14.931592 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.932085 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:14.932096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:14.933945 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
W20260926 12:29:14.934221 16784 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:14.928976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:52003 (local address 127.7.99.62:43399)
0926 12:29:14.929120 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:14.929127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:14.929145 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:29:14.929192 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:14.929196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:14.929268 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:14.929354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:14.929841 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.930425 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.931184 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:14.931431 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:14.931527 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:14.934104 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:52003: BlockingRecv error: recv got EOF from 127.7.99.3:52003 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:15.587002 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:15.581671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:42645)
0926 12:29:15.581925 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:15.581939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:15.582028 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:15.582245 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:15.582248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:15.582495 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:15.582594 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.582600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.583368 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.583370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.583892 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.583898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.584344 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.584347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.584420 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.584895 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:15.584909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:15.586889 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260926 12:29:15.587142 16788 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:15.581758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42645 (local address 127.7.99.62:43399)
0926 12:29:15.581997 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:15.582000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:15.582013 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:15.582104 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:15.582106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:15.582146 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:15.582213 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:15.582721 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.583282 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.583988 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.584243 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.584325 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.587013 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:42645: BlockingRecv error: recv got EOF from 127.7.99.1:42645 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:15.753111 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:15.747859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:49633)
0926 12:29:15.748121 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:15.748135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:15.748253 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:15.748466 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:15.748468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:15.748647 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:15.748737 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.748741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.749454 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.749456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.750023 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.750030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.750459 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.750462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.750532 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.751125 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:15.751136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:15.753002 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260926 12:29:15.753284 16788 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:15.747945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:49633 (local address 127.7.99.62:43399)
0926 12:29:15.748088 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:15.748091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:15.748208 (+ 117us) server_negotiation.cc:407] Connection header received
0926 12:29:15.748320 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:15.748322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:15.748361 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:15.748441 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:15.748864 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.749356 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.750138 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.750352 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.750440 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.753127 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:49633: BlockingRecv error: recv got EOF from 127.7.99.2:49633 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:15 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:15.940794 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:15.934953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:52453)
0926 12:29:15.935192 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:15.935213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:15.935340 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:15.935633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:15.935636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:15.935890 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:15.936032 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.936063 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.936948 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.936950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.937597 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:15.937606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.938150 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.938157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:15.938219 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.938752 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:15.938793 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:15.940689 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
W20260926 12:29:15.940968 16788 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:15.934995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:52453 (local address 127.7.99.62:43399)
0926 12:29:15.935162 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:15.935167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:15.935317 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:29:15.935443 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:15.935448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:15.935507 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:15.935590 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:15.936242 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.936831 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.937729 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:15.938032 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:15.938107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:15.940845 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:52453: BlockingRecv error: recv got EOF from 127.7.99.3:52453 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:16.592914 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:16.587705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:58969)
0926 12:29:16.587975 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:16.587988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:16.588078 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:16.588348 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:16.588350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:16.588528 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:16.588625 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.588630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.589320 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.589322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.589887 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.589894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.590275 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.590277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.590334 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.590941 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:16.590955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:16.592800 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
W20260926 12:29:16.593043 16792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:16.587789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:58969 (local address 127.7.99.62:43399)
0926 12:29:16.588017 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:16.588020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:16.588032 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:29:16.588184 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:16.588186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:16.588229 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:16.588311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:16.588719 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.589232 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.589981 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.590187 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.590258 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.592920 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:58969: BlockingRecv error: recv got EOF from 127.7.99.1:58969 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:16.761579 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:16.753877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:57841)
0926 12:29:16.754218 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:16.754234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:16.754483 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:16.754790 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:16.754793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:16.755036 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:16.755195 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.755202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.756395 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.756402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.757422 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.757432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.758071 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.758074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.758179 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.758691 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:16.758705 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:16.761468 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":67,"threads_started":1}
W20260926 12:29:16.761801 16792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:16.753930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:57841 (local address 127.7.99.62:43399)
0926 12:29:16.754322 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:16.754328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:16.754346 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:29:16.754599 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:16.754605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:16.754665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:16.754755 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:16.755311 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.756287 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.757551 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.757961 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.758055 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.761673 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:57841: BlockingRecv error: recv got EOF from 127.7.99.2:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":282}
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:16 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:16.946360 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:16.941496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:41359)
0926 12:29:16.941742 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:16.941755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:16.941842 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:16.942082 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:16.942084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:16.942252 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:16.942341 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.942346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.943073 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.943075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.943606 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:16.943612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.944035 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.944037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:16.944091 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.944588 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:16.944599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:16.946250 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260926 12:29:16.946542 16792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:16.941582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:41359 (local address 127.7.99.62:43399)
0926 12:29:16.941804 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:16.941807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:16.941821 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:29:16.941936 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:16.941938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:16.941985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:16.942051 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:16.942433 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.942984 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.943701 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:16.943941 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:16.944018 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:16.946387 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:41359: BlockingRecv error: recv got EOF from 127.7.99.3:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:17.603129 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:17.593727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:49663)
0926 12:29:17.593992 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:17.594005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:17.594093 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:17.594314 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:17.594316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:17.594492 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:17.594589 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.594594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.595298 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.595300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.595858 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.595865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.596367 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.596371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.596444 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.601082 (+ 4638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:17.601095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:17.602956 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
W20260926 12:29:17.603222 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:17.593802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49663 (local address 127.7.99.62:43399)
0926 12:29:17.594030 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:17.594033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:17.594050 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:29:17.594161 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:17.594163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:17.594203 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:17.594277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:17.594688 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.595204 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.595952 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.596220 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.596325 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.603090 (+ 6765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:49663: BlockingRecv error: recv got EOF from 127.7.99.1:49663 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:17.767935 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:17.762448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:47135)
0926 12:29:17.762694 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:17.762707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:17.762789 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:17.763064 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:17.763066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:17.763234 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:17.763327 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.763332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.764104 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.764106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.764711 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.764720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.765178 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.765180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.765236 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.765838 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:17.765849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:17.767756 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20260926 12:29:17.768074 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:17.762587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:47135 (local address 127.7.99.62:43399)
0926 12:29:17.762731 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:17.762736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:17.762750 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:29:17.762901 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:17.762904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:17.762954 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:17.763026 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:17.763448 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.764015 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.764836 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.765064 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.765158 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.767952 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:47135: BlockingRecv error: recv got EOF from 127.7.99.2:47135 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:17 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:17.952673 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:17.947120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:57951)
0926 12:29:17.947399 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:17.947411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:17.947499 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:17.947797 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:17.947799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:17.947975 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:17.948065 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.948070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.948864 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.948866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.949574 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:17.949583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.950067 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.950069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:17.950134 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.950613 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:17.950624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:17.952566 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
W20260926 12:29:17.952926 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:17.947191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:57951 (local address 127.7.99.62:43399)
0926 12:29:17.947372 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:17.947380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:17.947462 (+ 82us) server_negotiation.cc:407] Connection header received
0926 12:29:17.947646 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:17.947649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:17.947696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:17.947769 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:17.948175 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.948767 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.949681 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:17.949934 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:17.950038 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:17.952777 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:57951: BlockingRecv error: recv got EOF from 127.7.99.3:57951 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:18.609215 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:18.604008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:49101)
0926 12:29:18.604292 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:18.604305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:18.604403 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:18.604628 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:18.604630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:18.604803 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:18.604899 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.604904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.605622 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.605624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.606167 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.606173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.606625 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.606627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.606686 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.607221 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:18.607232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:18.609088 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
W20260926 12:29:18.609349 16800 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:18.604063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49101 (local address 127.7.99.62:43399)
0926 12:29:18.604401 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:18.604405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:18.604420 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:29:18.604481 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:18.604483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:18.604523 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:18.604597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:18.605004 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.605531 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.606294 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.606522 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.606610 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.609219 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:49101: BlockingRecv error: recv got EOF from 127.7.99.1:49101 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:18.773902 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:18.768656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:44605)
0926 12:29:18.768909 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:18.768923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:18.769020 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:18.769241 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:18.769243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:18.769433 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:18.769524 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.769529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.770210 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.770213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.770787 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.770795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.771187 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.771189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.771267 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.771727 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:18.771737 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:18.773726 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
W20260926 12:29:18.773938 16800 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:18.768700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:44605 (local address 127.7.99.62:43399)
0926 12:29:18.768845 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:18.768849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:18.768990 (+ 141us) server_negotiation.cc:407] Connection header received
0926 12:29:18.769096 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:18.769098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:18.769144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:18.769213 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:18.769613 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.770118 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.770881 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.771093 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.771165 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.773842 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:44605: BlockingRecv error: recv got EOF from 127.7.99.2:44605 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:18 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.3@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:18.959055 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:18.953494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.3:46899)
0926 12:29:18.953820 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:18.953832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:18.953982 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:18.954236 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:18.954239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:18.954472 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:18.954558 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.954563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.955338 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.955343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.955962 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:18.955972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.956480 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.956484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:18.956570 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.957117 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:18.957131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:18.958879 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
W20260926 12:29:18.959097 16800 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:18.953608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:46899 (local address 127.7.99.62:43399)
0926 12:29:18.953769 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:18.953773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:18.953909 (+ 136us) server_negotiation.cc:407] Connection header received
0926 12:29:18.954091 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:18.954094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:18.954137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:18.954219 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:18.954665 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.955235 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.956078 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:18.956370 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:18.956479 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:18.958994 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.3:46899: BlockingRecv error: recv got EOF from 127.7.99.3:46899 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 12:29:19 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:19 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.1@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:19.615151 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:19.610207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.1:48187)
0926 12:29:19.610490 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:19.610521 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:19.610600 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:19.610811 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:19.610813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:19.610997 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:19.611093 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:19.611098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.611811 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.611813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:19.612378 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:19.612386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.612768 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.612770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:19.612825 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:19.613274 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:19.613284 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:19.615020 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
W20260926 12:29:19.615540 16804 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:19.610283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:48187 (local address 127.7.99.62:43399)
0926 12:29:19.610558 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:19.610562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:19.610575 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:29:19.610682 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:19.610684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:19.610727 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:19.610801 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:19.611186 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.611724 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.612468 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.612677 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.612750 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:19.615153 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.1:48187: BlockingRecv error: recv got EOF from 127.7.99.1:48187 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
Sep 26 12:29:19 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:29:19 dist-test-slave-3jw2 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.99.2@KRBTEST.COM for oryx/127.7.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:29:19.780440 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:19.774806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43399 (local address 127.7.99.2:34561)
0926 12:29:19.775145 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:19.775162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:19.775299 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:19.775576 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:19.775578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:19.775782 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:19.775925 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:19.775931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.776765 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.776771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:19.777335 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:19.777343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.777803 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.777807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:19.777909 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:19.778376 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:19.778387 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:19.780232 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.99.62:43399: Server oryx/127.7.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
W20260926 12:29:19.780494 16804 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:19.774940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:34561 (local address 127.7.99.62:43399)
0926 12:29:19.775126 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:19.775131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:19.775252 (+ 121us) server_negotiation.cc:407] Connection header received
0926 12:29:19.775408 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:19.775412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:19.775459 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:19.775556 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:19.776044 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.776657 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.777453 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:19.777692 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:19.777801 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:19.780383 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.99.2:34561: BlockingRecv error: recv got EOF from 127.7.99.2:34561 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 12:29:19.859596 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16329
I20260926 12:29:19.866051 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16479
I20260926 12:29:19.872069 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16614
I20260926 12:29:19.877801 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16259
2026-09-26T12:29:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17883 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T12:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:19Z Disabled control of system clock
I20260926 12:29:19.907972 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:42867
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:29:20.014555 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:20.014808 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:20.014842 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:20.018697 16809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:20.018745 16809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:20.018760 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:20.018776 16809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:20.018790 16809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:20.023553 16809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:42867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:20.024513 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:20.025770 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:20.034107 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.034226 16809 server_base.cc:1034] running on GCE node
W20260926 12:29:20.034166 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.034096 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.034852 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:20.035889 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:20.037073 16809 hybrid_clock.cc:648] HybridClock initialized: now 1790425760037035 us; error 50 us; skew 500 ppm
I20260926 12:29:20.038782 16809 webserver.cc:460] Webserver started at http://127.7.99.62:36949/ using document root <none> and password file <none>
I20260926 12:29:20.039332 16809 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:20.039417 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:20.039644 16809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:20.041613 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a05a25cc66349af8929247a08df163fc"
server_key_iv: "352252540897e278548c1d9246252fe7"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.042160 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a05a25cc66349af8929247a08df163fc"
server_key_iv: "352252540897e278548c1d9246252fe7"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.045533 16809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:29:20.047717 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.048878 16809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:20.049009 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a05a25cc66349af8929247a08df163fc"
server_key_iv: "352252540897e278548c1d9246252fe7"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.049118 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:20.058790 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:20.059346 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:20.059525 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:20.065896 16809 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:42867
I20260926 12:29:20.065913 16875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:42867 every 8 connection(s)
I20260926 12:29:20.066738 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:29:20.069847 16876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:20.073550 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16809
I20260926 12:29:20.073666 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:29:20.073969 7564 external_mini_cluster.cc:1428] Setting key 8a700fe64c1eb0d2b8b86d8aa7db49d6
I20260926 12:29:20.075572 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152: Bootstrap starting.
I20260926 12:29:20.077670 16876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:20.078372 16876 log.cc:824] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:20.080006 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152: No bootstrap required, opened a new log
I20260926 12:29:20.082860 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.075305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.7.99.62:42867)
0926 12:29:20.075951 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.075967 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.075999 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:29:20.076662 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.076678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.077044 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.077391 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.077843 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.078826 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.079686 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.080248 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.080418 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.081255 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:20.081260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:20.081271 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:20.081637 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:20.081686 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:20.081704 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:20.081809 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:20.081955 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 12:29:20.082184 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260926 12:29:20.083586 16876 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } }
I20260926 12:29:20.083822 16876 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:20.083894 16876 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7f87e2b14c4eeea9c37a0fe8b84152, State: Initialized, Role: FOLLOWER
I20260926 12:29:20.084338 16876 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } }
I20260926 12:29:20.084482 16876 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:29:20.084542 16876 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:29:20.084630 16876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:20.085516 16876 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } }
I20260926 12:29:20.085781 16876 leader_election.cc:302] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7f87e2b14c4eeea9c37a0fe8b84152; no voters:
I20260926 12:29:20.086011 16876 leader_election.cc:288] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:29:20.086119 16881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:20.086387 16881 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [term 1 LEADER]: Becoming Leader. State: Replica: ba7f87e2b14c4eeea9c37a0fe8b84152, State: Running, Role: LEADER
I20260926 12:29:20.086686 16881 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } }
I20260926 12:29:20.087188 16876 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:29:20.088359 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } } }
I20260926 12:29:20.088402 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7f87e2b14c4eeea9c37a0fe8b84152. Latest consensus state: current_term: 1 leader_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7f87e2b14c4eeea9c37a0fe8b84152" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 42867 } } }
I20260926 12:29:20.088541 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:20.088562 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:20.088965 16890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:29:20.091909 16890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:29:20.097249 16890 catalog_manager.cc:1347] Generated new cluster ID: 93b540287e69405a881e7f6eee003344
I20260926 12:29:20.097326 16890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:29:20.105928 16890 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:29:20.107028 16890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:29:20.114487 16890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7f87e2b14c4eeea9c37a0fe8b84152: Generated new TSK 0
I20260926 12:29:20.115200 16890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:29:20.118464 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--builtin_ntp_servers=127.7.99.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 12:29:20.227329 16900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:20.227594 16900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:20.227679 16900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:20.231690 16900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:20.231770 16900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:20.231854 16900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:29:20.236475 16900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:20.237500 16900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:20.238842 16900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:20.248472 16908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.248497 16905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.248478 16906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.248927 16900 server_base.cc:1034] running on GCE node
I20260926 12:29:20.249320 16900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:20.249912 16900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:20.251103 16900 hybrid_clock.cc:648] HybridClock initialized: now 1790425760251086 us; error 36 us; skew 500 ppm
I20260926 12:29:20.252898 16900 webserver.cc:460] Webserver started at http://127.7.99.1:39917/ using document root <none> and password file <none>
I20260926 12:29:20.253458 16900 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:20.253548 16900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:20.253786 16900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:20.255697 16900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "cf9c4527e0c34133a8ff665f24cdf202"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a6a41ba954464936b9c9fe21c46070fc"
server_key_iv: "ae3808b198b7c0d17bd7a9ccaab3854e"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.256304 16900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "cf9c4527e0c34133a8ff665f24cdf202"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a6a41ba954464936b9c9fe21c46070fc"
server_key_iv: "ae3808b198b7c0d17bd7a9ccaab3854e"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.259737 16900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:29:20.262338 16914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.263459 16900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:20.263590 16900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "cf9c4527e0c34133a8ff665f24cdf202"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "a6a41ba954464936b9c9fe21c46070fc"
server_key_iv: "ae3808b198b7c0d17bd7a9ccaab3854e"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.263701 16900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:20.282135 16900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:20.282805 16900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:20.283023 16900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:20.283591 16900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:20.284572 16900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:20.284643 16900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.284713 16900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:20.284761 16900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.293376 16900 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:44431
I20260926 12:29:20.293402 17027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:44431 every 8 connection(s)
I20260926 12:29:20.294261 16900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:29:20.294924 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 16900
I20260926 12:29:20.295030 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:29:20.295329 7564 external_mini_cluster.cc:1428] Setting key 8c8e31837e6c631c93e3d40bee4a5ad6
I20260926 12:29:20.297592 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--builtin_ntp_servers=127.7.99.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:29:20.304358 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.296262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:39319 (local address 127.7.99.62:42867)
0926 12:29:20.296394 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.296399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.297708 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 12:29:20.299652 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.299656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.299707 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.299814 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.300757 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.301360 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.302461 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.302691 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.302798 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.303855 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:20.303858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:20.303861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:20.303885 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:20.303903 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:20.303911 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:20.303988 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:20.304145 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:29:20.304221 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:29:20.305152 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.296706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:42867 (local address 127.7.99.1:39319)
0926 12:29:20.297530 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:20.297585 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:20.299448 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:20.299985 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:20.299992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:20.300074 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:20.300619 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.300632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.301479 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.301489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.302330 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.302338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.302836 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.302840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.303011 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.303605 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:20.303607 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:20.303645 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:20.303705 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:20.303989 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:20.304000 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:20.304008 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 12:29:20.304135 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 12:29:20.304428 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260926 12:29:20.306411 17028 heartbeater.cc:347] Connected to a master server at 127.7.99.62:42867
I20260926 12:29:20.306623 17028 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:20.307081 17028 heartbeater.cc:511] Master 127.7.99.62:42867 requested a full tablet report, sending...
I20260926 12:29:20.308519 16840 ts_manager.cc:194] Registered new tserver with Master: cf9c4527e0c34133a8ff665f24cdf202 (127.7.99.1:44431)
I20260926 12:29:20.309585 16840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.1:39319
W20260926 12:29:20.406296 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:20.406571 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:20.406642 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:20.410595 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:20.410679 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:20.410769 17031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:29:20.415450 17031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:20.416539 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:20.417937 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:29:20.427495 17031 server_base.cc:1034] running on GCE node
W20260926 12:29:20.427573 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.427640 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.427592 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.428399 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:20.429117 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:20.430320 17031 hybrid_clock.cc:648] HybridClock initialized: now 1790425760430285 us; error 52 us; skew 500 ppm
I20260926 12:29:20.432374 17031 webserver.cc:460] Webserver started at http://127.7.99.2:44571/ using document root <none> and password file <none>
I20260926 12:29:20.432928 17031 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:20.432986 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:20.433178 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:20.435108 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "935864b78cf34a53a8618dc83d7c43f5"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "f620a2bb164b703b24ab16876eb96ec1"
server_key_iv: "1cf6957b00f5a9a8a6fad0030efaff5c"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.435602 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "935864b78cf34a53a8618dc83d7c43f5"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "f620a2bb164b703b24ab16876eb96ec1"
server_key_iv: "1cf6957b00f5a9a8a6fad0030efaff5c"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.439148 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:29:20.441783 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.442952 17031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:29:20.443045 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "935864b78cf34a53a8618dc83d7c43f5"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "f620a2bb164b703b24ab16876eb96ec1"
server_key_iv: "1cf6957b00f5a9a8a6fad0030efaff5c"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.443177 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:20.455740 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:20.456468 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:20.456652 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:20.457195 17031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:20.458078 17031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:20.458122 17031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.458185 17031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:20.458221 17031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.466549 17031 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:39883
I20260926 12:29:20.466615 17158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:39883 every 8 connection(s)
I20260926 12:29:20.467432 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:29:20.473991 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17031
I20260926 12:29:20.474115 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:29:20.474406 7564 external_mini_cluster.cc:1428] Setting key dc0a88913c615a110e813cad449344eb
I20260926 12:29:20.476392 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--builtin_ntp_servers=127.7.99.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:29:20.476619 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.469241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:35049 (local address 127.7.99.62:42867)
0926 12:29:20.469398 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.469402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.470363 (+ 961us) server_negotiation.cc:407] Connection header received
0926 12:29:20.471773 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.471776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.471831 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.471920 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.472803 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.473330 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.474534 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.474791 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.474914 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.476080 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:20.476083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:20.476086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:20.476097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:20.476124 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:20.476132 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:20.476241 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:20.476420 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:29:20.476490 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:29:20.477439 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.469627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:42867 (local address 127.7.99.2:35049)
0926 12:29:20.470216 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:20.470255 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:20.471535 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:20.472090 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:20.472096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:20.472198 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:20.472658 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.472670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.473450 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.473455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.474419 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.474428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.474984 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.474989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.475185 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.475811 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:20.475814 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:20.475866 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:20.475924 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:20.476258 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:20.476286 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:20.476296 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 12:29:20.476420 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 12:29:20.476693 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260926 12:29:20.478852 17159 heartbeater.cc:347] Connected to a master server at 127.7.99.62:42867
I20260926 12:29:20.479079 17159 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:20.479552 17159 heartbeater.cc:511] Master 127.7.99.62:42867 requested a full tablet report, sending...
I20260926 12:29:20.480535 16840 ts_manager.cc:194] Registered new tserver with Master: 935864b78cf34a53a8618dc83d7c43f5 (127.7.99.2:39883)
I20260926 12:29:20.481086 16840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.2:35049
W20260926 12:29:20.587445 17162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:20.587715 17162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:20.587795 17162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:20.592000 17162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:20.592080 17162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:20.592227 17162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:29:20.596932 17162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:20.597965 17162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:20.599290 17162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:20.608474 17168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.608551 17162 server_base.cc:1034] running on GCE node
W20260926 12:29:20.608498 17170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.608485 17167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.609117 17162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:20.609722 17162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:20.610909 17162 hybrid_clock.cc:648] HybridClock initialized: now 1790425760610895 us; error 26 us; skew 500 ppm
I20260926 12:29:20.612797 17162 webserver.cc:460] Webserver started at http://127.7.99.3:43819/ using document root <none> and password file <none>
I20260926 12:29:20.613348 17162 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:20.613436 17162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:20.613659 17162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:20.615593 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "43c2ca461e6c4b6bbbf02965aa5f4d43"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "b6c09440af18f4d16ca7bbaf68bbd513"
server_key_iv: "e3c1586228dc4d9258c9c6f1929a8883"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.616227 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "43c2ca461e6c4b6bbbf02965aa5f4d43"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "b6c09440af18f4d16ca7bbaf68bbd513"
server_key_iv: "e3c1586228dc4d9258c9c6f1929a8883"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.619541 17162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 12:29:20.621948 17176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.623162 17162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:20.623303 17162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "43c2ca461e6c4b6bbbf02965aa5f4d43"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "b6c09440af18f4d16ca7bbaf68bbd513"
server_key_iv: "e3c1586228dc4d9258c9c6f1929a8883"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.623409 17162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:20.636085 17162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:20.636709 17162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:20.636895 17162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:20.637421 17162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:20.638267 17162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:20.638334 17162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.638398 17162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:20.638450 17162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.647370 17162 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:42475
I20260926 12:29:20.647416 17289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:42475 every 8 connection(s)
I20260926 12:29:20.648329 17162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:29:20.653141 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17162
I20260926 12:29:20.653227 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:29:20.653527 7564 external_mini_cluster.cc:1428] Setting key 9ceabe6a8532defb468d91854291ff39
I20260926 12:29:20.657832 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.650033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:35203 (local address 127.7.99.62:42867)
0926 12:29:20.650158 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.650162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.651196 (+ 1034us) server_negotiation.cc:407] Connection header received
0926 12:29:20.652798 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.652801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.652860 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.652992 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.654028 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.654776 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.656132 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.656430 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.656508 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.657463 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:20.657466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:20.657468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:20.657478 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:20.657494 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:20.657502 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:20.657558 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:20.657686 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 12:29:20.657729 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:29:20.658681 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.650397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:42867 (local address 127.7.99.3:35203)
0926 12:29:20.651046 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:20.651084 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:20.652573 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:20.653167 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:20.653177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:20.653308 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:20.653880 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.653899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.654920 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.654924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.656021 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:20.656030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:20.656552 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:20.656554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:20.656696 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:20.657238 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:20.657240 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:20.657279 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:20.657336 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:20.657574 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:20.657580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:20.657584 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:29:20.657671 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:29:20.657929 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":100,"threads_started":1}
I20260926 12:29:20.659727 17290 heartbeater.cc:347] Connected to a master server at 127.7.99.62:42867
I20260926 12:29:20.660076 17290 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:20.660586 17290 heartbeater.cc:511] Master 127.7.99.62:42867 requested a full tablet report, sending...
I20260926 12:29:20.661502 16840 ts_manager.cc:194] Registered new tserver with Master: 43c2ca461e6c4b6bbbf02965aa5f4d43 (127.7.99.3:42475)
I20260926 12:29:20.662004 16840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.3:35203
I20260926 12:29:20.667150 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:29:20.669848 16879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:20.668926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.7.99.62:42867)
0926 12:29:20.669061 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.669065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.669231 (+ 166us) server_negotiation.cc:407] Connection header received
0926 12:29:20.669371 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.669377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.669419 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.669514 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.669739 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46986: BlockingRecv error: recv got EOF from 127.0.0.1:46986 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:29:20.669982 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16900
I20260926 12:29:20.675905 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17031
I20260926 12:29:20.681252 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17162
I20260926 12:29:20.686355 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 16809
2026-09-26T12:29:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (807 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T12:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:20Z Disabled control of system clock
I20260926 12:29:20.721868 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:33915
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:29:20.830039 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:20.830281 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:20.830317 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:20.834287 17301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:20.834345 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:20.834373 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:20.834388 17301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:20.834422 17301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:20.839242 17301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:33915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:20.840330 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:20.841663 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:20.849956 17306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.850109 17301 server_base.cc:1034] running on GCE node
W20260926 12:29:20.849959 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:20.850041 17307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:20.850809 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:20.851851 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:20.853006 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790425760852994 us; error 23 us; skew 500 ppm
I20260926 12:29:20.854733 17301 webserver.cc:460] Webserver started at http://127.7.99.62:43907/ using document root <none> and password file <none>
I20260926 12:29:20.855324 17301 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:20.855384 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:20.855616 17301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:20.858246 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "84731258603c4eb7a8f705b99f1d67d0"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "2f5d8f3eab8978d836ccb4ecdf1736cf"
server_key_iv: "52f250444a3116997a99567b3f24198f"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.858801 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "84731258603c4eb7a8f705b99f1d67d0"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "2f5d8f3eab8978d836ccb4ecdf1736cf"
server_key_iv: "52f250444a3116997a99567b3f24198f"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.862355 17301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:29:20.864816 17315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:20.866022 17301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:20.866151 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "84731258603c4eb7a8f705b99f1d67d0"
format_stamp: "Formatted at 2026-09-26 12:29:20 on dist-test-slave-3jw2"
server_key: "2f5d8f3eab8978d836ccb4ecdf1736cf"
server_key_iv: "52f250444a3116997a99567b3f24198f"
server_key_version: "encryptionkey@0"
I20260926 12:29:20.866317 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:20.870723 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:20.871253 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:20.871379 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:20.878109 17301 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:33915
I20260926 12:29:20.878113 17367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:33915 every 8 connection(s)
I20260926 12:29:20.878983 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:29:20.881996 17368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:20.887270 17368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0: Bootstrap starting.
I20260926 12:29:20.887302 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17301
I20260926 12:29:20.887434 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:29:20.887698 7564 external_mini_cluster.cc:1428] Setting key 0577a51481a352f21ce69ec6f53d1ce5
I20260926 12:29:20.889879 17368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:20.890619 17368 log.cc:824] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:20.892812 17368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0: No bootstrap required, opened a new log
I20260926 12:29:20.893268 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:20.889023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.7.99.62:33915)
0926 12:29:20.889635 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:20.889651 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:20.889685 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:29:20.890352 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:20.890369 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:20.890713 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:20.891085 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:20.891839 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:20.891850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:20.891862 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:20.891872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:20.891925 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:20.891941 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:20.892058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:20.892208 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 12:29:20.892490 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":143,"threads_started":1}
I20260926 12:29:20.896770 17368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } }
I20260926 12:29:20.897006 17368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:20.897081 17368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84731258603c4eb7a8f705b99f1d67d0, State: Initialized, Role: FOLLOWER
I20260926 12:29:20.897490 17368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } }
I20260926 12:29:20.897634 17368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:29:20.897718 17368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:29:20.897836 17368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:20.898712 17368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } }
I20260926 12:29:20.899061 17368 leader_election.cc:302] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84731258603c4eb7a8f705b99f1d67d0; no voters:
I20260926 12:29:20.899360 17368 leader_election.cc:288] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:29:20.899466 17373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:20.899712 17373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [term 1 LEADER]: Becoming Leader. State: Replica: 84731258603c4eb7a8f705b99f1d67d0, State: Running, Role: LEADER
I20260926 12:29:20.900079 17373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } }
I20260926 12:29:20.900614 17368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:29:20.901790 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84731258603c4eb7a8f705b99f1d67d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } } }
I20260926 12:29:20.901773 17375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84731258603c4eb7a8f705b99f1d67d0. Latest consensus state: current_term: 1 leader_uuid: "84731258603c4eb7a8f705b99f1d67d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84731258603c4eb7a8f705b99f1d67d0" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33915 } } }
I20260926 12:29:20.901911 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:20.901911 17375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:20.902354 17382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:29:20.905416 17382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:29:20.910861 17382 catalog_manager.cc:1347] Generated new cluster ID: 8181c36770704c1da04499241b65992d
I20260926 12:29:20.910929 17382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:29:20.919551 17382 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:29:20.919838 17382 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:29:20.925794 17382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84731258603c4eb7a8f705b99f1d67d0: Generated new TSK 0
I20260926 12:29:20.926391 17382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:29:20.933955 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--builtin_ntp_servers=127.7.99.20:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:29:21.041225 17392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:21.041450 17392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:21.041487 17392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:21.045246 17392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:21.045292 17392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:21.045390 17392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:29:21.049897 17392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:21.050896 17392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:21.052227 17392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:21.061466 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:21.061466 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:21.061561 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:21.061901 17392 server_base.cc:1034] running on GCE node
I20260926 12:29:21.062237 17392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:21.062825 17392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:21.063974 17392 hybrid_clock.cc:648] HybridClock initialized: now 1790425761063945 us; error 43 us; skew 500 ppm
I20260926 12:29:21.065735 17392 webserver.cc:460] Webserver started at http://127.7.99.1:38993/ using document root <none> and password file <none>
I20260926 12:29:21.066285 17392 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:21.066344 17392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:21.066497 17392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:21.068501 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "00564f0b584245788c0be66334cfa375"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "45043a59f814ebfbc59f40e89a18ffe9"
server_key_iv: "4140032e720905eac00dc3253bd94982"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.069056 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "00564f0b584245788c0be66334cfa375"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "45043a59f814ebfbc59f40e89a18ffe9"
server_key_iv: "4140032e720905eac00dc3253bd94982"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.072703 17392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:29:21.074985 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.076135 17392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:21.076282 17392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "00564f0b584245788c0be66334cfa375"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "45043a59f814ebfbc59f40e89a18ffe9"
server_key_iv: "4140032e720905eac00dc3253bd94982"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.076364 17392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:21.080588 17392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:21.081104 17392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:21.081228 17392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:21.081658 17392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:21.082438 17392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:21.082482 17392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.082516 17392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:21.082537 17392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.090745 17392 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:40039
I20260926 12:29:21.090801 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:40039 every 8 connection(s)
I20260926 12:29:21.091538 17392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:29:21.097108 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.093267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42579 (local address 127.7.99.62:33915)
0926 12:29:21.093396 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:21.093417 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:21.094409 (+ 992us) server_negotiation.cc:407] Connection header received
0926 12:29:21.095590 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:21.095593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:21.095625 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:21.095695 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:21.096752 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:21.096755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:21.096757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:21.096767 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:21.096784 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:21.096792 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:21.096863 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:21.096974 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 12:29:21.097009 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:29:21.097919 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.093619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33915 (local address 127.7.99.1:42579)
0926 12:29:21.094257 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:21.094294 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:21.095415 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:21.095852 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:21.095858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:21.095938 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:21.096518 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:21.096520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:21.096555 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:21.096619 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:21.096862 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:21.096867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:21.096871 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:29:21.096991 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 12:29:21.097198 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260926 12:29:21.099063 17520 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33915
I20260926 12:29:21.099291 17520 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:21.099665 17520 heartbeater.cc:511] Master 127.7.99.62:33915 requested a full tablet report, sending...
I20260926 12:29:21.099854 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17392
I20260926 12:29:21.099959 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:29:21.100332 7564 external_mini_cluster.cc:1428] Setting key 6f2e1073d23ec1d1efb56ac2b032d5c3
I20260926 12:29:21.101326 17332 ts_manager.cc:194] Registered new tserver with Master: 00564f0b584245788c0be66334cfa375 (127.7.99.1:40039)
I20260926 12:29:21.102319 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--builtin_ntp_servers=127.7.99.20:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:29:21.209519 17523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:21.209796 17523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:21.209862 17523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:21.213769 17523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:21.213853 17523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:21.213946 17523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:29:21.218561 17523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:21.219584 17523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:21.220952 17523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:21.230182 17528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:21.230221 17529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:21.230343 17523 server_base.cc:1034] running on GCE node
W20260926 12:29:21.230182 17531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:21.230871 17523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:21.231491 17523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:21.232672 17523 hybrid_clock.cc:648] HybridClock initialized: now 1790425761232659 us; error 27 us; skew 500 ppm
I20260926 12:29:21.234414 17523 webserver.cc:460] Webserver started at http://127.7.99.2:35019/ using document root <none> and password file <none>
I20260926 12:29:21.234972 17523 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:21.235064 17523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:21.235282 17523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:21.237315 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "75c7b44e16c7482a9dd31eaccd5c8d80"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "fb77dda6cf7ab353467002570bea6141"
server_key_iv: "e91fce1caf40136c1004a4b561064f4f"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.237866 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "75c7b44e16c7482a9dd31eaccd5c8d80"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "fb77dda6cf7ab353467002570bea6141"
server_key_iv: "e91fce1caf40136c1004a4b561064f4f"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.241319 17523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:29:21.243649 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.244874 17523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:21.245015 17523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "75c7b44e16c7482a9dd31eaccd5c8d80"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "fb77dda6cf7ab353467002570bea6141"
server_key_iv: "e91fce1caf40136c1004a4b561064f4f"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.245119 17523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:21.250705 17523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:21.251276 17523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:21.251462 17523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:21.251968 17523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:21.252875 17523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:21.252945 17523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.253005 17523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:21.253036 17523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.261474 17523 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:38611
I20260926 12:29:21.261497 17650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:38611 every 8 connection(s)
I20260926 12:29:21.262282 17523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:29:21.268098 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.263988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:41425 (local address 127.7.99.62:33915)
0926 12:29:21.264118 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:21.264124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:21.265161 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 12:29:21.266425 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:21.266428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:21.266471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:21.266552 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:21.267623 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:21.267627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:21.267629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:21.267641 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:21.267661 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:21.267670 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:21.267771 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:21.267922 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 12:29:21.267966 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:29:21.268317 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17523
I20260926 12:29:21.268448 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:29:21.268744 7564 external_mini_cluster.cc:1428] Setting key d15df78ce55099796c5a287d21c04b6b
I20260926 12:29:21.269042 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.264395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33915 (local address 127.7.99.2:41425)
0926 12:29:21.265014 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:21.265051 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:21.266211 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:21.266712 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:21.266718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:21.266799 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:21.267372 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:21.267374 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:21.267411 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:21.267474 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:21.267766 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:21.267775 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:21.267782 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:29:21.267958 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 12:29:21.268205 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
I20260926 12:29:21.270430 17651 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33915
I20260926 12:29:21.270685 17651 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:21.270720 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--builtin_ntp_servers=127.7.99.20:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 12:29:21.271095 17651 heartbeater.cc:511] Master 127.7.99.62:33915 requested a full tablet report, sending...
I20260926 12:29:21.272089 17331 ts_manager.cc:194] Registered new tserver with Master: 75c7b44e16c7482a9dd31eaccd5c8d80 (127.7.99.2:38611)
W20260926 12:29:21.375492 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:21.375736 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:21.375773 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:21.379688 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:21.379738 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:21.379827 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:29:21.384406 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:21.385375 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:21.386718 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:21.396060 17659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:21.396183 17660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:21.396072 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:21.396358 17654 server_base.cc:1034] running on GCE node
I20260926 12:29:21.396775 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:21.397410 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:21.398593 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790425761398562 us; error 48 us; skew 500 ppm
I20260926 12:29:21.400485 17654 webserver.cc:460] Webserver started at http://127.7.99.3:36545/ using document root <none> and password file <none>
I20260926 12:29:21.401103 17654 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:21.401192 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:21.401422 17654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:21.403565 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "29cff6bbaf7b41f7bf91e47569aefe39"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "1f073ff41590a5d924b8a282014f392b"
server_key_iv: "c01846b6a71280222f78fb62110cb8a3"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.404100 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "29cff6bbaf7b41f7bf91e47569aefe39"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "1f073ff41590a5d924b8a282014f392b"
server_key_iv: "c01846b6a71280222f78fb62110cb8a3"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.407655 17654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:29:21.410066 17668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.411270 17654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:21.411410 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "29cff6bbaf7b41f7bf91e47569aefe39"
format_stamp: "Formatted at 2026-09-26 12:29:21 on dist-test-slave-3jw2"
server_key: "1f073ff41590a5d924b8a282014f392b"
server_key_iv: "c01846b6a71280222f78fb62110cb8a3"
server_key_version: "encryptionkey@0"
I20260926 12:29:21.411520 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:21.415769 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:21.416401 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:21.416566 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:21.417079 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:21.417901 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:21.417970 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.418035 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:21.418087 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:21.426424 17654 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:36155
I20260926 12:29:21.426502 17781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:36155 every 8 connection(s)
I20260926 12:29:21.427316 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:29:21.433295 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.429117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:39179 (local address 127.7.99.62:33915)
0926 12:29:21.429309 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:21.429314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:21.430369 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 12:29:21.431700 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:21.431703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:21.431736 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:21.431798 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:21.432922 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:21.432925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:21.432927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:29:21.432937 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:21.432955 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:29:21.432963 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:21.433036 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:21.433162 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 12:29:21.433197 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:29:21.434440 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:21.429570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33915 (local address 127.7.99.3:39179)
0926 12:29:21.430199 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:21.430254 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:21.431501 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:21.431942 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:21.431949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:21.432044 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:21.432660 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:29:21.432662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:21.432704 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:29:21.432779 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:21.433043 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:21.433049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:21.433053 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:29:21.433184 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 12:29:21.433372 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260926 12:29:21.435467 17782 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33915
I20260926 12:29:21.435648 17782 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:21.435976 17782 heartbeater.cc:511] Master 127.7.99.62:33915 requested a full tablet report, sending...
I20260926 12:29:21.436810 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17654
I20260926 12:29:21.436925 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:29:21.436893 17331 ts_manager.cc:194] Registered new tserver with Master: 29cff6bbaf7b41f7bf91e47569aefe39 (127.7.99.3:36155)
I20260926 12:29:21.437199 7564 external_mini_cluster.cc:1428] Setting key 352d15de3fba8ff30e9288a82b651301
I20260926 12:29:21.439010 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:29:21.441620 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:29:21.440799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.7.99.62:33915)
0926 12:29:21.440900 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:21.440903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:21.441132 (+ 229us) server_negotiation.cc:407] Connection header received
0926 12:29:21.441174 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:21.441177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:21.441207 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:21.441286 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:21.441451 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55816: BlockingRecv error: recv got EOF from 127.0.0.1:55816 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:29:21.441843 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17392
I20260926 12:29:21.447714 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17523
I20260926 12:29:21.453193 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17654
I20260926 12:29:21.458209 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17301
2026-09-26T12:29:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (771 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:29:21 dist-test-slave-3jw2 krb5kdc[17792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:29:21 dist-test-slave-3jw2 krb5kdc[17792](info): set up 2 sockets
Sep 26 12:29:21 dist-test-slave-3jw2 krb5kdc[17792](info): commencing operation
krb5kdc: starting...
W20260926 12:29:22.525439 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:29:22 dist-test-slave-3jw2 krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:29:22.643838 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44011
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:22.755884 17805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:22.756109 17805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:22.756147 17805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:22.760005 17805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:29:22.760054 17805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:22.760073 17805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:22.760089 17805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:29:22.760103 17805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:29:22.765159 17805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:22.766033 17805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:22.766921 17805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:22.774895 17811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:22.774914 17813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:22.775082 17805 server_base.cc:1034] running on GCE node
W20260926 12:29:22.774899 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:22.775674 17805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:22.776777 17805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:22.777958 17805 hybrid_clock.cc:648] HybridClock initialized: now 1790425762777946 us; error 23 us; skew 500 ppm
Sep 26 12:29:22 dist-test-slave-3jw2 krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425762, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:22.786367 17805 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:29:22.787245 17805 webserver.cc:460] Webserver started at http://127.7.99.62:37169/ using document root <none> and password file <none>
I20260926 12:29:22.787782 17805 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:22.787879 17805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:22.788105 17805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:22.790055 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "2abe7a3438704591908c3c6877f92046"
format_stamp: "Formatted at 2026-09-26 12:29:22 on dist-test-slave-3jw2"
server_key: "01fb8c61f2a90bd4dc8848bb791abdd8"
server_key_iv: "c61fbefa343238e96ac4fa984cec0968"
server_key_version: "encryptionkey@0"
I20260926 12:29:22.790606 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "2abe7a3438704591908c3c6877f92046"
format_stamp: "Formatted at 2026-09-26 12:29:22 on dist-test-slave-3jw2"
server_key: "01fb8c61f2a90bd4dc8848bb791abdd8"
server_key_iv: "c61fbefa343238e96ac4fa984cec0968"
server_key_version: "encryptionkey@0"
I20260926 12:29:22.794130 17805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:29:22.796321 17820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:22.797466 17805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:22.797613 17805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "2abe7a3438704591908c3c6877f92046"
format_stamp: "Formatted at 2026-09-26 12:29:22 on dist-test-slave-3jw2"
server_key: "01fb8c61f2a90bd4dc8848bb791abdd8"
server_key_iv: "c61fbefa343238e96ac4fa984cec0968"
server_key_version: "encryptionkey@0"
I20260926 12:29:22.797747 17805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:22.825223 17805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:22.828549 17805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:22.828789 17805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:22.835532 17805 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44011
I20260926 12:29:22.835809 17872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44011 every 8 connection(s)
I20260926 12:29:22.836522 17805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:29:22.839184 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:22.839560 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17805
I20260926 12:29:22.839754 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:29:22.840080 7564 external_mini_cluster.cc:1428] Setting key 2bd1a64bd88321fef6a26291533097f2
I20260926 12:29:22.845299 17873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: Bootstrap starting.
I20260926 12:29:22.847918 17873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:22.848822 17873 log.cc:824] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:29:22 dist-test-slave-3jw2 krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:22.850652 17873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: No bootstrap required, opened a new log
I20260926 12:29:22.854019 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:22.841476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.7.99.62:44011)
0926 12:29:22.841893 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:22.841917 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:22.841963 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:29:22.842603 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:22.842619 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:22.842945 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:22.843232 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:22.844119 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:22.845176 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:22.846298 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:22.846763 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:22.846889 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:22.849632 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:22.849637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:22.849649 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:22.849674 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:22.851385 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:22.851819 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:22.851824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:22.851828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:22.851938 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:22.852241 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:22.852244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:22.852246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:22.852608 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:22.852800 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:22.853053 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 12:29:22.853314 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260926 12:29:22.854178 17873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } }
I20260926 12:29:22.854422 17873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:22.854504 17873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abe7a3438704591908c3c6877f92046, State: Initialized, Role: FOLLOWER
I20260926 12:29:22.854992 17873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } }
I20260926 12:29:22.855163 17873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:29:22.855228 17873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:29:22.855317 17873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:22.856436 17873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } }
I20260926 12:29:22.856720 17873 leader_election.cc:302] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abe7a3438704591908c3c6877f92046; no voters:
I20260926 12:29:22.856949 17873 leader_election.cc:288] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:29:22.857057 17878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:22.857309 17878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [term 1 LEADER]: Becoming Leader. State: Replica: 2abe7a3438704591908c3c6877f92046, State: Running, Role: LEADER
I20260926 12:29:22.857633 17878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } }
I20260926 12:29:22.858089 17873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:29:22.859444 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abe7a3438704591908c3c6877f92046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } } }
I20260926 12:29:22.859562 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:22.859768 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abe7a3438704591908c3c6877f92046. Latest consensus state: current_term: 1 leader_uuid: "2abe7a3438704591908c3c6877f92046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abe7a3438704591908c3c6877f92046" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44011 } } }
I20260926 12:29:22.860090 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046 [sys.catalog]: This master's current role is: LEADER
I20260926 12:29:22.860369 17893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:29:22.863241 17893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:29:22.868527 17893 catalog_manager.cc:1347] Generated new cluster ID: f884bcebfb3541e8bb290ec444748f3f
I20260926 12:29:22.868593 17893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:29:22.890146 17893 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:29:22.891139 17893 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:29:22.897080 17893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: Generated new TSK 0
I20260926 12:29:22.897769 17893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:29:22.923509 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:23.031705 17899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:23.031997 17899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:23.032071 17899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:23.036021 17899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:23.036108 17899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:23.036214 17899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:29:23.041400 17899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:23.042411 17899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:23.043392 17899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:23.052672 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.052654 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.052685 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:23.052847 17899 server_base.cc:1034] running on GCE node
I20260926 12:29:23.053356 17899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:23.053967 17899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:23.055120 17899 hybrid_clock.cc:648] HybridClock initialized: now 1790425763055092 us; error 41 us; skew 500 ppm
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:23.057775 17899 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:29:23.058712 17899 webserver.cc:460] Webserver started at http://127.7.99.1:37625/ using document root <none> and password file <none>
I20260926 12:29:23.059306 17899 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:23.059410 17899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:23.059612 17899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:23.061636 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "dcb5c083caac40179f0537fe46a30b2b"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "d75c45d461f620f1c3fce6c29ad6856c"
server_key_iv: "4d3ec13141eb74c66bec11ad24736d28"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.062206 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "dcb5c083caac40179f0537fe46a30b2b"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "d75c45d461f620f1c3fce6c29ad6856c"
server_key_iv: "4d3ec13141eb74c66bec11ad24736d28"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.065758 17899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:29:23.068011 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.069272 17899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:23.069401 17899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "dcb5c083caac40179f0537fe46a30b2b"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "d75c45d461f620f1c3fce6c29ad6856c"
server_key_iv: "4d3ec13141eb74c66bec11ad24736d28"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.069509 17899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:23.091430 17899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:23.094168 17899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:23.094362 17899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:23.094951 17899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:23.095728 17899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:23.095799 17899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.095872 17899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:23.095916 17899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.104813 17899 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:38365
I20260926 12:29:23.104833 18027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:38365 every 8 connection(s)
I20260926 12:29:23.105671 17899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:29:23.109499 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 17899
I20260926 12:29:23.109591 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:29:23.109894 7564 external_mini_cluster.cc:1428] Setting key fd766ffe4bdc0adbe9d6cce8b0fcaf46
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:29:23.119503 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.107464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:41173 (local address 127.7.99.62:44011)
0926 12:29:23.107593 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.107596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.108522 (+ 926us) server_negotiation.cc:407] Connection header received
0926 12:29:23.109927 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.109931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.109986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.110072 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:23.111528 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.112051 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.113202 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.113434 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.113544 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.115958 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:23.115962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:23.115965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:23.115996 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:23.117521 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.118019 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.118024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.118026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.118104 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.118448 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.118453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.118454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.118634 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:23.118739 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.119206 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.119334 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:29:23.120458 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.107915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44011 (local address 127.7.99.1:41173)
0926 12:29:23.108384 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.108425 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.109727 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.110274 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.110282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.110762 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:23.111393 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.111409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.112194 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.112197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.113071 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.113079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.113576 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.113580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.113757 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.114336 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:23.114352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:23.115803 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:23.117650 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.117655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.117669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.117920 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.118195 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.118199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.118201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.118369 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.118737 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:23.118743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:23.118983 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.119170 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.119526 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:29:23.121790 18028 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44011
I20260926 12:29:23.122036 18028 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:23.122581 18028 heartbeater.cc:511] Master 127.7.99.62:44011 requested a full tablet report, sending...
I20260926 12:29:23.124101 17837 ts_manager.cc:194] Registered new tserver with Master: dcb5c083caac40179f0537fe46a30b2b (127.7.99.1:38365)
I20260926 12:29:23.125309 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:41173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:29:23.135058 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:23.241400 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:23.241627 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:23.241667 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:23.245486 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:23.245534 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:23.245607 18033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:29:23.250757 18033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:23.251655 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:23.252635 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:23.263057 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.263082 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.263057 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:23.263290 18033 server_base.cc:1034] running on GCE node
I20260926 12:29:23.263751 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:23.264398 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:23.265564 18033 hybrid_clock.cc:648] HybridClock initialized: now 1790425763265533 us; error 42 us; skew 500 ppm
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:23.268111 18033 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:29:23.269021 18033 webserver.cc:460] Webserver started at http://127.7.99.2:33505/ using document root <none> and password file <none>
I20260926 12:29:23.269557 18033 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:23.269616 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:23.269853 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:23.271799 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "c8b634ca0a2045909596146d7c15320e"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "a3eb8ce176756d3dd4aa15f211adcdd5"
server_key_iv: "91247afa506edfc873987b5e228ab49e"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.272372 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "c8b634ca0a2045909596146d7c15320e"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "a3eb8ce176756d3dd4aa15f211adcdd5"
server_key_iv: "91247afa506edfc873987b5e228ab49e"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.275924 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:29:23.278326 18048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.279475 18033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:29:23.279592 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "c8b634ca0a2045909596146d7c15320e"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "a3eb8ce176756d3dd4aa15f211adcdd5"
server_key_iv: "91247afa506edfc873987b5e228ab49e"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.279690 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:23.289425 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:23.291975 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:23.292155 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:23.292726 18033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:23.293476 18033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:23.293520 18033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.293583 18033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:23.293619 18033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.302654 18033 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:44309
I20260926 12:29:23.302711 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:44309 every 8 connection(s)
I20260926 12:29:23.303522 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:29:23.310585 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 18033
I20260926 12:29:23.310714 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:29:23.311005 7564 external_mini_cluster.cc:1428] Setting key 89c1a6cb5c5f4717fe803fd83b87e7ff
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:23.317577 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.305287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:43771 (local address 127.7.99.62:44011)
0926 12:29:23.305423 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.305426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.306212 (+ 786us) server_negotiation.cc:407] Connection header received
0926 12:29:23.307521 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.307524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.307579 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.307685 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:23.308797 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.309329 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.310455 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.310705 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.310821 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.313526 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:23.313529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:23.313531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:23.313554 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:23.315280 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.315906 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.315908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.315910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.315987 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.316418 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.316421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.316422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.316613 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:23.316697 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.317347 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.317459 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:29:23.318825 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.305655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44011 (local address 127.7.99.2:43771)
0926 12:29:23.306071 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.306109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.307310 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.307847 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.307853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.308204 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:23.308657 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.308669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.309451 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.309454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.310297 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.310305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.310870 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.310874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.311035 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.311622 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:23.311646 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:23.313359 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:23.315410 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.315414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.315426 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.315798 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.316095 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.316101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.316106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.316311 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.316725 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:23.316734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:23.317083 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.317301 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.317661 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
I20260926 12:29:23.320055 18162 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44011
I20260926 12:29:23.320325 18162 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:23.320811 18162 heartbeater.cc:511] Master 127.7.99.62:44011 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:29:23.321827 17837 ts_manager.cc:194] Registered new tserver with Master: c8b634ca0a2045909596146d7c15320e (127.7.99.2:44309)
I20260926 12:29:23.322448 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:43771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:29:23.338558 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:29:23.445327 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:29:23.445566 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:29:23.445606 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:29:23.449503 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:29:23.449550 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:29:23.449644 18167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:29:23.454630 18167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:29:23.455555 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:29:23.456528 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:29:23.465718 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.465727 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:29:23.465799 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:29:23.465868 18167 server_base.cc:1034] running on GCE node
I20260926 12:29:23.466361 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:29:23.466902 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:29:23.468055 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790425763468038 us; error 28 us; skew 500 ppm
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:29:23.470661 18167 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:29:23.471443 18167 webserver.cc:460] Webserver started at http://127.7.99.3:42347/ using document root <none> and password file <none>
I20260926 12:29:23.471979 18167 fs_manager.cc:359] Metadata directory not provided
I20260926 12:29:23.472067 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:29:23.472334 18167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:29:23.474413 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "7736bb9eff0d4015a17a880cac0e9b66"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "3efa273f5a078421df57a8d1a01fb200"
server_key_iv: "bca78f969c4c9c9ac83c74a49248c813"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.474972 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "7736bb9eff0d4015a17a880cac0e9b66"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "3efa273f5a078421df57a8d1a01fb200"
server_key_iv: "bca78f969c4c9c9ac83c74a49248c813"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.478690 18167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:29:23.481029 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.482157 18167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:29:23.482308 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "7736bb9eff0d4015a17a880cac0e9b66"
format_stamp: "Formatted at 2026-09-26 12:29:23 on dist-test-slave-3jw2"
server_key: "3efa273f5a078421df57a8d1a01fb200"
server_key_iv: "bca78f969c4c9c9ac83c74a49248c813"
server_key_version: "encryptionkey@0"
I20260926 12:29:23.482434 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:29:23.498276 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:29:23.500958 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:29:23.501168 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:29:23.501693 18167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:29:23.502470 18167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:29:23.502540 18167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.502609 18167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:29:23.502657 18167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:29:23.511932 18167 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:36973
I20260926 12:29:23.511953 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:36973 every 8 connection(s)
I20260926 12:29:23.512822 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:29:23.514472 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 18167
I20260926 12:29:23.514591 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:29:23.514886 7564 external_mini_cluster.cc:1428] Setting key 14d00d15702dae0bf57d82fb8a35982a
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425763, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:29:23.526280 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.514537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:39981 (local address 127.7.99.62:44011)
0926 12:29:23.514698 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.514702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.515651 (+ 949us) server_negotiation.cc:407] Connection header received
0926 12:29:23.517348 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.517351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.517412 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.517517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:23.518619 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.519148 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.520286 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.520512 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.520620 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.522882 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:23.522886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:23.522889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:23.522920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:23.524340 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.524814 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.524819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.524821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.524908 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.525234 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.525239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.525241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.525419 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:23.525524 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.525978 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.526098 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:29:23.527033 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.515001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44011 (local address 127.7.99.3:39981)
0926 12:29:23.515500 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.515554 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.517147 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.517686 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.517693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.518037 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:29:23.518487 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.518499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.519266 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.519269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.520144 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.520153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.520652 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.520656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.520826 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.521374 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:29:23.521389 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:29:23.522709 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:29:23.524455 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.524461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.524473 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.524713 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.525009 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:29:23.525012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:29:23.525014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:29:23.525143 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:29:23.525520 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:29:23.525526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:29:23.525754 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.525939 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.526255 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":160,"threads_started":1}
I20260926 12:29:23.528343 18296 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44011
I20260926 12:29:23.528599 18296 heartbeater.cc:464] Registering TS with master...
I20260926 12:29:23.529114 18296 heartbeater.cc:511] Master 127.7.99.62:44011 requested a full tablet report, sending...
I20260926 12:29:23.530021 17837 ts_manager.cc:194] Registered new tserver with Master: 7736bb9eff0d4015a17a880cac0e9b66 (127.7.99.3:36973)
I20260926 12:29:23.530658 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:39981
I20260926 12:29:23.540534 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:29:23.548935 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.542354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.7.99.62:44011)
0926 12:29:23.542479 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.542482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.542640 (+ 158us) server_negotiation.cc:407] Connection header received
0926 12:29:23.542733 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.542735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.542769 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.542835 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:29:23.543466 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.543946 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.544756 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.544958 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.545032 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.546171 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:29:23.546174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:29:23.546176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:29:23.546196 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:29:23.547563 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.547900 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.547902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.547903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.547970 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:29:23.548242 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:29:23.548244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:29:23.548245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:29:23.548458 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:29:23.548558 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.548740 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.548823 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:29:23.551551 17837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:29:23.554581 17837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:29:23.568538 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.562832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.7.99.1:38365)
0926 12:29:23.563619 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.563630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.563652 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:29:23.563727 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.563732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.563947 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.564106 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:23.564443 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.565595 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.567220 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.567872 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.567988 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.568127 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.568294 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.568386 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":154,"threads_started":1}
I20260926 12:29:23.568575 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.562686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38365 (local address 127.0.0.1:34766)
0926 12:29:23.562989 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.563032 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.563148 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.564098 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.564104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.564127 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:23.564293 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.564300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.565723 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.565726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.567097 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.567108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.567987 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.567991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.568090 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.568123 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.568260 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.568380 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":63,"thread_start_us":87,"threads_started":1}
I20260926 12:29:23.568607 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.562962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36973 (local address 127.0.0.1:42944)
0926 12:29:23.563263 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.563278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.563366 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.564340 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.564343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.564360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:23.564452 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.564458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.565521 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.565524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.567097 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.567108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.568182 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.568186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.568278 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.568302 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.568355 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.568423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":98,"thread_start_us":81,"threads_started":1}
I20260926 12:29:23.568917 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.563668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.7.99.2:44309)
0926 12:29:23.564144 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.564181 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.564502 (+ 321us) server_negotiation.cc:407] Connection header received
0926 12:29:23.564574 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.564578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.564748 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.564901 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:23.565119 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.566065 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.567266 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.568024 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.568138 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.568290 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.568683 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.568780 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":137,"threads_started":1}
I20260926 12:29:23.569351 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.563468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.7.99.3:36973)
0926 12:29:23.563923 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.563934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.563953 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:29:23.564022 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.564028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.564223 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.564359 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:23.564544 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.565399 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.567239 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.568024 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.569028 (+ 1004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.569150 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.569189 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.569251 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":94,"threads_started":1}
I20260926 12:29:23.569588 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.563382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44309 (local address 127.0.0.1:35230)
0926 12:29:23.564307 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.564320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.564414 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.564897 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.564900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.564918 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:23.565005 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.565010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.566211 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.566214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.567143 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.567152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.568349 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.568353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.568453 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.568475 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.568541 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.568594 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"spinlock_wait_cycles":2432,"thread_start_us":88,"threads_started":1}
I20260926 12:29:23.571789 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 0db439c95cd944799e70155e624324c0 (DEFAULT_TABLE table=test-table [id=ee43cf602cb845d8b39b381d97483cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:23.571789 17962 tablet_service.cc:1467] Processing CreateTablet for tablet 0db439c95cd944799e70155e624324c0 (DEFAULT_TABLE table=test-table [id=ee43cf602cb845d8b39b381d97483cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:23.572913 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db439c95cd944799e70155e624324c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:23.572918 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db439c95cd944799e70155e624324c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:23.574687 18096 tablet_service.cc:1467] Processing CreateTablet for tablet 0db439c95cd944799e70155e624324c0 (DEFAULT_TABLE table=test-table [id=ee43cf602cb845d8b39b381d97483cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:29:23.575505 18096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db439c95cd944799e70155e624324c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:29:23.580024 18313 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Bootstrap starting.
I20260926 12:29:23.580809 18314 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Bootstrap starting.
I20260926 12:29:23.582136 18315 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Bootstrap starting.
I20260926 12:29:23.582733 18314 tablet_bootstrap.cc:654] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:23.582784 18313 tablet_bootstrap.cc:654] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:23.584056 18314 log.cc:824] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:23.584296 18313 log.cc:824] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:23.584738 18315 tablet_bootstrap.cc:654] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Neither blocks nor log segments found. Creating new log.
I20260926 12:29:23.585517 18315 log.cc:824] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:29:23.586037 18314 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: No bootstrap required, opened a new log
I20260926 12:29:23.586153 18313 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: No bootstrap required, opened a new log
I20260926 12:29:23.586171 18314 ts_tablet_manager.cc:1397] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:29:23.586275 18313 ts_tablet_manager.cc:1397] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 12:29:23.589165 18314 raft_consensus.cc:348] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.589223 18313 raft_consensus.cc:348] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.589396 18314 raft_consensus.cc:374] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:23.589442 18314 raft_consensus.cc:729] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7736bb9eff0d4015a17a880cac0e9b66, State: Initialized, Role: FOLLOWER
I20260926 12:29:23.589454 18313 raft_consensus.cc:374] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:23.589507 18313 raft_consensus.cc:729] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb5c083caac40179f0537fe46a30b2b, State: Initialized, Role: FOLLOWER
I20260926 12:29:23.589751 18315 tablet_bootstrap.cc:492] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: No bootstrap required, opened a new log
I20260926 12:29:23.589911 18315 ts_tablet_manager.cc:1397] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260926 12:29:23.589871 18314 consensus_queue.cc:260] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.589912 18313 consensus_queue.cc:260] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.590821 18314 ts_tablet_manager.cc:1428] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:29:23.590819 18313 ts_tablet_manager.cc:1428] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 12:29:23.591143 18028 heartbeater.cc:503] Master 127.7.99.62:44011 was elected leader, sending a full tablet report...
I20260926 12:29:23.591308 18296 heartbeater.cc:503] Master 127.7.99.62:44011 was elected leader, sending a full tablet report...
I20260926 12:29:23.593143 18315 raft_consensus.cc:348] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.593345 18315 raft_consensus.cc:374] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:29:23.593432 18315 raft_consensus.cc:729] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b634ca0a2045909596146d7c15320e, State: Initialized, Role: FOLLOWER
I20260926 12:29:23.593839 18315 consensus_queue.cc:260] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.594432 18162 heartbeater.cc:503] Master 127.7.99.62:44011 was elected leader, sending a full tablet report...
I20260926 12:29:23.594724 18315 ts_tablet_manager.cc:1428] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260926 12:29:23.606814 18029 tablet.cc:2369] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:29:23.763827 18297 tablet.cc:2369] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:29:23.769537 18320 raft_consensus.cc:482] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:29:23.769708 18320 raft_consensus.cc:504] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.770797 18320 leader_election.cc:288] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8b634ca0a2045909596146d7c15320e (127.7.99.2:44309), 7736bb9eff0d4015a17a880cac0e9b66 (127.7.99.3:36973)
I20260926 12:29:23.775108 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.771163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:37413 (local address 127.7.99.2:44309)
0926 12:29:23.771289 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.771294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.771520 (+ 226us) server_negotiation.cc:407] Connection header received
0926 12:29:23.771620 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.771627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.771695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.771778 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:23.772068 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.772726 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774030 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.774452 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.774593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.774911 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.774986 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:29:23.775120 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.771168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49431 (local address 127.7.99.3:36973)
0926 12:29:23.771289 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.771294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.771697 (+ 403us) server_negotiation.cc:407] Connection header received
0926 12:29:23.771832 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.771835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.771891 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.771967 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:29:23.772270 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.772983 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774124 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.774633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774728 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.774784 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.774956 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.775024 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:29:23.775120 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.771070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44309 (local address 127.7.99.1:37413)
0926 12:29:23.771428 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.771440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.771536 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.771808 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.771810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.771827 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:23.771961 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.771967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.772808 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.772811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.773919 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.773928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.774663 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.774765 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.774787 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.774870 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.774953 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":27,"thread_start_us":90,"threads_started":1}
I20260926 12:29:23.775126 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.771070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36973 (local address 127.7.99.1:49431)
0926 12:29:23.771586 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.771599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:29:23.771711 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:29:23.771991 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:29:23.771993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:29:23.772035 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:29:23.772141 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.772147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.773089 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.773091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.773988 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:29:23.774001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.774736 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.774738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:29:23.774815 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.774832 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:29:23.774926 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:29:23.774986 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":46,"thread_start_us":108,"threads_started":1}
I20260926 12:29:23.775830 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db439c95cd944799e70155e624324c0" candidate_uuid: "dcb5c083caac40179f0537fe46a30b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b634ca0a2045909596146d7c15320e" is_pre_election: true
I20260926 12:29:23.775970 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db439c95cd944799e70155e624324c0" candidate_uuid: "dcb5c083caac40179f0537fe46a30b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7736bb9eff0d4015a17a880cac0e9b66" is_pre_election: true
I20260926 12:29:23.776139 18116 raft_consensus.cc:2454] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcb5c083caac40179f0537fe46a30b2b in term 0.
I20260926 12:29:23.776307 18250 raft_consensus.cc:2454] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcb5c083caac40179f0537fe46a30b2b in term 0.
I20260926 12:29:23.776651 17915 leader_election.cc:302] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8b634ca0a2045909596146d7c15320e, dcb5c083caac40179f0537fe46a30b2b; no voters:
I20260926 12:29:23.776888 18320 raft_consensus.cc:2781] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:29:23.776966 18320 raft_consensus.cc:482] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:29:23.777009 18320 raft_consensus.cc:3037] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:23.777980 18320 raft_consensus.cc:504] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.778406 18320 leader_election.cc:288] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [CANDIDATE]: Term 1 election: Requested vote from peers c8b634ca0a2045909596146d7c15320e (127.7.99.2:44309), 7736bb9eff0d4015a17a880cac0e9b66 (127.7.99.3:36973)
I20260926 12:29:23.778684 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db439c95cd944799e70155e624324c0" candidate_uuid: "dcb5c083caac40179f0537fe46a30b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b634ca0a2045909596146d7c15320e"
I20260926 12:29:23.778820 18116 raft_consensus.cc:3037] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:23.778806 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db439c95cd944799e70155e624324c0" candidate_uuid: "dcb5c083caac40179f0537fe46a30b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7736bb9eff0d4015a17a880cac0e9b66"
I20260926 12:29:23.778949 18250 raft_consensus.cc:3037] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:29:23.779822 18116 raft_consensus.cc:2454] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcb5c083caac40179f0537fe46a30b2b in term 1.
I20260926 12:29:23.779971 18250 raft_consensus.cc:2454] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcb5c083caac40179f0537fe46a30b2b in term 1.
I20260926 12:29:23.780076 17915 leader_election.cc:302] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8b634ca0a2045909596146d7c15320e, dcb5c083caac40179f0537fe46a30b2b; no voters:
I20260926 12:29:23.780261 18320 raft_consensus.cc:2781] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:29:23.780463 18320 raft_consensus.cc:686] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 1 LEADER]: Becoming Leader. State: Replica: dcb5c083caac40179f0537fe46a30b2b, State: Running, Role: LEADER
I20260926 12:29:23.780781 18320 consensus_queue.cc:237] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } }
I20260926 12:29:23.783778 17835 catalog_manager.cc:5495] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to dcb5c083caac40179f0537fe46a30b2b (127.7.99.1). New cstate: current_term: 1 leader_uuid: "dcb5c083caac40179f0537fe46a30b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5c083caac40179f0537fe46a30b2b" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 } health_report { overall_health: UNKNOWN } } }
I20260926 12:29:23.800314 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.796828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.7.99.1:38365)
0926 12:29:23.796948 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:29:23.796953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:29:23.797037 (+ 84us) server_negotiation.cc:407] Connection header received
0926 12:29:23.797221 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:29:23.797223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:29:23.797272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:29:23.797386 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:29:23.797622 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.798106 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.798960 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:29:23.799172 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:29:23.799278 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:29:23.799568 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:29:23.799942 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:29:23.800069 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:29:23.800110 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 12:29:23.800160 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 12:29:23.804593 18163 tablet.cc:2369] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:29:23.805907 18116 raft_consensus.cc:1261] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 1 FOLLOWER]: Refusing update from remote peer dcb5c083caac40179f0537fe46a30b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:23.805907 18250 raft_consensus.cc:1261] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 1 FOLLOWER]: Refusing update from remote peer dcb5c083caac40179f0537fe46a30b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:29:23.806538 18324 consensus_queue.cc:1035] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b634ca0a2045909596146d7c15320e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:23.806757 18320 consensus_queue.cc:1035] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7736bb9eff0d4015a17a880cac0e9b66" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:29:23.813560 18329 mvcc.cc:205] Tried to move back new op lower bound from 7333583928541679616 to 7333583928448536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333583928541679616}]
I20260926 12:29:23.814879 18328 mvcc.cc:205] Tried to move back new op lower bound from 7333583928541679616 to 7333583928448536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333583928541679616}]
I20260926 12:29:23.821616 17835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:23.821768 17835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:29:23.823930 17835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2abe7a3438704591908c3c6877f92046: Sending DeleteTablet for 3 replicas of tablet 0db439c95cd944799e70155e624324c0
I20260926 12:29:23.824615 17962 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db439c95cd944799e70155e624324c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:34766
I20260926 12:29:23.824683 18096 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db439c95cd944799e70155e624324c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:35230
I20260926 12:29:23.824819 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db439c95cd944799e70155e624324c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:29:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:42944
I20260926 12:29:23.825232 18338 tablet_replica.cc:330] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: stopping tablet replica
I20260926 12:29:23.825243 18336 tablet_replica.cc:330] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b: stopping tablet replica
I20260926 12:29:23.825486 18337 tablet_replica.cc:330] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: stopping tablet replica
I20260926 12:29:23.825536 18336 raft_consensus.cc:2229] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:29:23.825677 18338 raft_consensus.cc:2229] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:23.825712 18337 raft_consensus.cc:2229] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:29:23.825857 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17899
I20260926 12:29:23.825896 18336 raft_consensus.cc:2258] T 0db439c95cd944799e70155e624324c0 P dcb5c083caac40179f0537fe46a30b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:23.825974 18337 raft_consensus.cc:2258] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:23.826150 18338 raft_consensus.cc:2258] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:29:23.827729 18338 ts_tablet_manager.cc:1905] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:23.827987 18337 ts_tablet_manager.cc:1905] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:29:23.830334 18337 ts_tablet_manager.cc:1918] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:29:23.830461 18337 log.cc:1196] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/0db439c95cd944799e70155e624324c0
I20260926 12:29:23.830802 18337 ts_tablet_manager.cc:1939] T 0db439c95cd944799e70155e624324c0 P c8b634ca0a2045909596146d7c15320e: Deleting consensus metadata
I20260926 12:29:23.831586 18338 ts_tablet_manager.cc:1918] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:29:23.831707 18338 log.cc:1196] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/0db439c95cd944799e70155e624324c0
I20260926 12:29:23.831979 17821 catalog_manager.cc:4841] TS c8b634ca0a2045909596146d7c15320e (127.7.99.2:44309): tablet 0db439c95cd944799e70155e624324c0 (table test-table [id=ee43cf602cb845d8b39b381d97483cbc]) successfully deleted
I20260926 12:29:23.832105 18338 ts_tablet_manager.cc:1939] T 0db439c95cd944799e70155e624324c0 P 7736bb9eff0d4015a17a880cac0e9b66: Deleting consensus metadata
I20260926 12:29:23.833381 17823 catalog_manager.cc:4841] TS 7736bb9eff0d4015a17a880cac0e9b66 (127.7.99.3:36973): tablet 0db439c95cd944799e70155e624324c0 (table test-table [id=ee43cf602cb845d8b39b381d97483cbc]) successfully deleted
W20260926 12:29:23.834769 17821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:38365 (error 108)
I20260926 12:29:23.834954 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 18033
I20260926 12:29:23.837332 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:29:23.837019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38365 (local address 127.0.0.1:34784)
0926 12:29:23.837164 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 12:29:23.837250 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:29:23.837531 17821 catalog_manager.cc:4584] TS dcb5c083caac40179f0537fe46a30b2b (127.7.99.1:38365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0db439c95cd944799e70155e624324c0: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38365: connect: Connection refused (error 111)
I20260926 12:29:23.840924 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 18167
I20260926 12:29:23.847951 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 17805
2026-09-26T12:29:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2391 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[18342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[18342](info): set up 2 sockets
Sep 26 12:29:23 dist-test-slave-3jw2 krb5kdc[18342](info): commencing operation
krb5kdc: starting...
W20260926 12:29:24.881366 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:29:24 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:29:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangeradmin_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangeradmin_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangerlookup_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangerlookup_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
I20260926 12:29:25.042014 7564 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT6LLMM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/postgres -l logfile start
2026-09-26 12:29:27.505 UTC [18372] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:29:27.505 UTC [18372] LOG: listening on IPv4 address "127.7.99.20", port 47077
2026-09-26 12:29:27.509 UTC [18372] LOG: listening on Unix socket "/tmp/.s.PGSQL.47077"
2026-09-26 12:29:27.530 UTC [18375] LOG: database system was shut down at 2026-09-26 12:29:26 UTC
2026-09-26 12:29:27.534 UTC [18372] LOG: database system is ready to accept connections
I20260926 12:29:28.493327 7564 mini_postgres.cc:96] Postgres bound to 47077
2026-09-26 12:29:28.499 UTC [18383] FATAL: database "slave" does not exist
127.7.99.20:47077 - accepting connections
I20260926 12:29:28.500116 7564 mini_ranger.cc:161] Starting Ranger...
I20260926 12:29:28.515043 7564 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:29:28.978878 7564 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:29:28.978967 7564 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin
2026-09-26 12:29:29,250 [I] DB FLAVOR :POSTGRES
2026-09-26 12:29:29,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:29:29,250 [I] Checking connection..
2026-09-26 12:29:29,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:29:29,511 [I] Checking connection passed.
2026-09-26 12:29:29,511 [I] --------- Verifying version history table ---------
2026-09-26 12:29:29,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:29:29,738 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:29:29,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:29:29,964 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:29:29,965 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:29:29,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:29:30.187 UTC [18447] WARNING: there is no transaction in progress
2026-09-26 12:29:30,193 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:29:30,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:29:30,438 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:29:30,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:29:30,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:29:30,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:29:30,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3jw2.c.gcp-upstream.internal','N') ;"
2026-09-26 12:29:31,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:29:31,423 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:29:31,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:29:31,689 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:29:31,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:29:31,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:29:32,156 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:29:32,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:29:33.125 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.140 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.156 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.170 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.188 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.207 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.266 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.272 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.285 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.295 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.304 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.316 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.325 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.331 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.338 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33.344 UTC [18555] WARNING: there is no transaction in progress
2026-09-26 12:29:33,940 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:29:33,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:29:34,186 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:29:34,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:29:34,421 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:29:34,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:29:34,661 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:29:34,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:29:34,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:29:35,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3jw2.c.gcp-upstream.internal';"
2026-09-26 12:29:35,333 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:29:35,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:29:35,550 [I] DB_PATCHES have already been applied
I20260926 12:29:35.556339 7564 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT6LLMM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:29:35.556419 7564 mini_ranger.cc:193] Using host: 127.7.99.20
I20260926 12:29:35.559770 7564 mini_ranger.cc:227] Ranger admin URL: http://127.7.99.20:57833
Sep 26, 2026 12:29:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:29:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:29:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:29:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:29:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:29:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:29:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:29:36.642241 7564 mini_ranger.cc:160] Time spent starting Ranger: real 8.142s user 0.004s sys 0.006s
Sep 26, 2026 12:29:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:29:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 12:29:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:29:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:29:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:29:58.358228 7564 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangerkms_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/rangerkms_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.99.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.99.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:29:58 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425798, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:29:58.442299 7564 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:29:58.452301 7564 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:29:59.284112 7564 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:29:59.284229 7564 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms
I20260926 12:29:59.284355 7564 mini_ranger_kms.cc:227] Using postgres at 127.7.99.20:47077
2026-09-26 12:29:59,468 [I] DB FLAVOR :POSTGRES
2026-09-26 12:29:59,469 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:29:59,469 [I] Checking connection
2026-09-26 12:29:59,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:29:59,706 [I] connection success
2026-09-26 12:29:59,706 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:29:59,706 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:29:59,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:29:59,953 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:29:59,954 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:29:59,954 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:29:59,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:47077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:30:00,198 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:30:00.641181 7564 mini_ranger_kms.cc:310] Created kms service
I20260926 12:30:00.844841 7564 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:30:00.886561 7564 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:30:00 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425798, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:01.287786 7564 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:30:01.287946 7564 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:30:01.287976 7564 mini_ranger_kms.cc:240] Using host: 127.7.99.20
I20260926 12:30:01.291435 7564 mini_ranger_kms.cc:276] Ranger KMS PID: 18735
I20260926 12:30:01.291574 7564 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.99.20:48747
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:30:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:30:03.585187 7564 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.143s user 0.000s sys 0.169s
I20260926 12:30:03.585328 7564 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:30:03.585445 7564 mini_ranger_kms.cc:401] 127.7.99.20:48747/kms/v1/keys
Sep 26 12:30:03 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790425803, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:30:04 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425803, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
Sep 26 12:30:04 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425803, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
Sep 26 12:30:06 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:30:06.500622 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:33365
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:33365
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:48747/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:06.624598 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:06.624850 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:06.624927 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:06.628966 18786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:06.629019 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:06.629038 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:06.629096 18786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:06.629122 18786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:06.634407 18786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.99.20:48747/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:33365
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:06.635392 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:06.636451 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:06.644531 18791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:06.644553 18794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:06.644636 18786 server_base.cc:1034] running on GCE node
W20260926 12:30:06.644587 18792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:06.645260 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:06.646330 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:06.647480 18786 hybrid_clock.cc:648] HybridClock initialized: now 1790425806647461 us; error 30 us; skew 500 ppm
Sep 26 12:30:06 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425806, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:06.657970 18786 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:30:06.658806 18786 webserver.cc:460] Webserver started at http://127.7.99.62:42441/ using document root <none> and password file <none>
I20260926 12:30:06.659296 18786 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:06.659345 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:06.659505 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:06 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425806, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:06.713630 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "33dc28b55ffc49dbb6df8921bade5490"
format_stamp: "Formatted at 2026-09-26 12:30:06 on dist-test-slave-3jw2"
server_key: "a078a8395cd1d30cff1dfd0420cd0315"
server_key_iv: "303b30deb477e627fa626d8f65c9feeb"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:06.714478 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "33dc28b55ffc49dbb6df8921bade5490"
format_stamp: "Formatted at 2026-09-26 12:30:06 on dist-test-slave-3jw2"
server_key: "a078a8395cd1d30cff1dfd0420cd0315"
server_key_iv: "303b30deb477e627fa626d8f65c9feeb"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:06.719483 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 12:30:06.737689 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:06.739120 18786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:30:06.739259 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "33dc28b55ffc49dbb6df8921bade5490"
format_stamp: "Formatted at 2026-09-26 12:30:06 on dist-test-slave-3jw2"
server_key: "a078a8395cd1d30cff1dfd0420cd0315"
server_key_iv: "303b30deb477e627fa626d8f65c9feeb"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:06.739398 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:06.758921 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:06.761582 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:06.761773 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:06.768560 18786 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:33365
I20260926 12:30:06.768584 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:33365 every 8 connection(s)
I20260926 12:30:06.769516 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:30:06.772053 18854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:06.777242 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: Bootstrap starting.
I20260926 12:30:06.778240 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 18786
I20260926 12:30:06.778376 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:30:06.778684 7564 external_mini_cluster.cc:1428] Setting key 8a52821376fbf926d537d72e0ae7293f
I20260926 12:30:06.780007 18854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:06.780757 18854 log.cc:824] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:06.782493 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: No bootstrap required, opened a new log
I20260926 12:30:06.785715 18854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } }
I20260926 12:30:06.785923 18854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:06.785979 18854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dc28b55ffc49dbb6df8921bade5490, State: Initialized, Role: FOLLOWER
I20260926 12:30:06.786405 18854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } }
I20260926 12:30:06.786552 18854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:30:06.786635 18854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:30:06.786756 18854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 12:30:06 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:06.787632 18854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } }
I20260926 12:30:06.788019 18854 leader_election.cc:302] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33dc28b55ffc49dbb6df8921bade5490; no voters:
I20260926 12:30:06.788343 18854 leader_election.cc:288] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:30:06.788450 18859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:30:06.788653 18859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [term 1 LEADER]: Becoming Leader. State: Replica: 33dc28b55ffc49dbb6df8921bade5490, State: Running, Role: LEADER
I20260926 12:30:06.788990 18859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } }
I20260926 12:30:06.789587 18854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:30:06.791045 18860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33dc28b55ffc49dbb6df8921bade5490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } } }
I20260926 12:30:06.791155 18860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:06.791992 18854 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:30:06.792179 18869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:30:06.794456 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:06.779736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.7.99.62:33365)
0926 12:30:06.780230 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:06.780249 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:06.780289 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:30:06.780930 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:06.780949 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:06.781340 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:06.781703 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:06.782463 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:06.783462 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:06.784360 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:06.784832 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:06.784996 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:06.787563 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:06.787569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:06.787590 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:06.787614 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:06.789643 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:06.790065 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:06.790071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:06.790077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:06.790194 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:06.791401 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:06.791404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:06.791406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:06.791873 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:06.792068 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:06.793333 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0926 12:30:06.793610 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
I20260926 12:30:06.795228 18861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33dc28b55ffc49dbb6df8921bade5490. Latest consensus state: current_term: 1 leader_uuid: "33dc28b55ffc49dbb6df8921bade5490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dc28b55ffc49dbb6df8921bade5490" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 33365 } } }
I20260926 12:30:06.795387 18861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:06.795395 18869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:30:06.805173 18869 catalog_manager.cc:1347] Generated new cluster ID: 76a82dacc15a4cc291ec2a7252f3c53f
I20260926 12:30:06.805275 18869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:30:06.812579 18869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:30:06.813969 18869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:30:06.821693 18869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: Generated new TSK 0
I20260926 12:30:06.822319 18869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:30:07.552466 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:48747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:07.700348 18899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:07.700723 18899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:07.700793 18899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:07.707054 18899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:07.707197 18899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:07.707321 18899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:30:07.715355 18899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.99.20:48747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:07.716903 18899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:07.718331 18899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:07.731380 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:07.731884 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:07.732429 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:07.733695 18899 server_base.cc:1034] running on GCE node
I20260926 12:30:07.734165 18899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:07.734925 18899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:07.736104 18899 hybrid_clock.cc:648] HybridClock initialized: now 1790425807736066 us; error 46 us; skew 500 ppm
Sep 26 12:30:07 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:07.739605 18899 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:30:07.740621 18899 webserver.cc:460] Webserver started at http://127.7.99.1:33287/ using document root <none> and password file <none>
I20260926 12:30:07.741281 18899 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:07.741348 18899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:07.741552 18899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:07 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:07.757810 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "e57a7f08f7914b589d90b4edcbbc1af5"
format_stamp: "Formatted at 2026-09-26 12:30:07 on dist-test-slave-3jw2"
server_key: "7fe59e6cb3fb2ef7c8e154a062947b16"
server_key_iv: "fe2d018c0151da5288fb5e2b3560f220"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:07.758529 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "e57a7f08f7914b589d90b4edcbbc1af5"
format_stamp: "Formatted at 2026-09-26 12:30:07 on dist-test-slave-3jw2"
server_key: "7fe59e6cb3fb2ef7c8e154a062947b16"
server_key_iv: "fe2d018c0151da5288fb5e2b3560f220"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:07.763513 18899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 12:30:07.783129 18918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:07.784791 18899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:30:07.784907 18899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "e57a7f08f7914b589d90b4edcbbc1af5"
format_stamp: "Formatted at 2026-09-26 12:30:07 on dist-test-slave-3jw2"
server_key: "7fe59e6cb3fb2ef7c8e154a062947b16"
server_key_iv: "fe2d018c0151da5288fb5e2b3560f220"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:07.785024 18899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:07.826306 18899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:07.830891 18899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:07.831192 18899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:07.831982 18899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:07.833165 18899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:07.833278 18899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:07.833411 18899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:07.833503 18899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:07.845654 18899 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:41179
I20260926 12:30:07.846401 19031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:41179 every 8 connection(s)
I20260926 12:30:07.846906 18899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:30:07 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:07.851405 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 18899
I20260926 12:30:07.851517 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:30:07.851833 7564 external_mini_cluster.cc:1428] Setting key 55cfb44699d104dde2cb7e8a48be513c
Sep 26 12:30:07 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:07.866880 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:07.850152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33365 (local address 127.7.99.1:44217)
0926 12:30:07.850726 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:07.850786 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:07.851953 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:07.852828 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:07.852837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:07.853216 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:07.853903 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:07.853929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:07.855109 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:07.855113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:07.856534 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:07.856547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:07.857406 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:07.857411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:07.857639 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:07.858420 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:07.858448 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:07.860346 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:07.863095 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:07.863103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:07.863125 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:07.863552 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:07.863898 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:07.863901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:07.863903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:07.864145 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:07.864721 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:07.864730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:07.865129 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 12:30:07.865363 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 12:30:07.865752 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":150,"threads_started":1}
I20260926 12:30:07.867738 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:07.849718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:44217 (local address 127.7.99.62:33365)
0926 12:30:07.850013 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:07.850019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:07.850897 (+ 878us) server_negotiation.cc:407] Connection header received
0926 12:30:07.852267 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:07.852271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:07.852335 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:07.852407 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:07.854110 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:07.854917 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:07.856697 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:07.857052 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:07.857160 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:07.860750 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:07.860755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:07.860758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:07.860797 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:07.862897 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:07.863687 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:07.863692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:07.863694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:07.863791 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:07.864286 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:07.864289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:07.864291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:07.864562 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:07.864661 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:07.867385 (+ 2724us) server_negotiation.cc:299] Negotiation successful
0926 12:30:07.867551 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
I20260926 12:30:07.869716 19032 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33365
I20260926 12:30:07.869989 19032 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:07.870663 19032 heartbeater.cc:511] Master 127.7.99.62:33365 requested a full tablet report, sending...
I20260926 12:30:07.872695 18818 ts_manager.cc:194] Registered new tserver with Master: e57a7f08f7914b589d90b4edcbbc1af5 (127.7.99.1:41179)
I20260926 12:30:07.874341 18818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:44217
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:30:07.903100 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:48747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:08.063792 19038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:08.064211 19038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:08.064287 19038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:08.070562 19038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:08.070642 19038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:08.070731 19038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:30:08.078891 19038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.99.20:48747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:08.080514 19038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:08.081897 19038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:08.094542 19047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:08.094547 19044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:08.094890 19045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:08.095595 19038 server_base.cc:1034] running on GCE node
I20260926 12:30:08.096136 19038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:08.096849 19038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:08.098156 19038 hybrid_clock.cc:648] HybridClock initialized: now 1790425808098123 us; error 47 us; skew 500 ppm
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:08.102010 19038 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:30:08.103205 19038 webserver.cc:460] Webserver started at http://127.7.99.2:41131/ using document root <none> and password file <none>
I20260926 12:30:08.103920 19038 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:08.104027 19038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:08.104364 19038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:08.119786 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "eb1664fee5bb4adfb9280ee8a4eb8150"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "77bb740b40e30269e535cb22a953f3b2"
server_key_iv: "acf49512c1cc87fcd7543115cee14b3e"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.120615 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "eb1664fee5bb4adfb9280ee8a4eb8150"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "77bb740b40e30269e535cb22a953f3b2"
server_key_iv: "acf49512c1cc87fcd7543115cee14b3e"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.125630 19038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 12:30:08.145817 19054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.147836 19038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:30:08.148093 19038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "eb1664fee5bb4adfb9280ee8a4eb8150"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "77bb740b40e30269e535cb22a953f3b2"
server_key_iv: "acf49512c1cc87fcd7543115cee14b3e"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.148375 19038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:08.187561 19038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:08.191290 19038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:08.191546 19038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:08.192338 19038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:08.193426 19038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:08.193536 19038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.193634 19038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:08.193699 19038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.207589 19038 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:45257
I20260926 12:30:08.208868 19038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:30:08.209473 19167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:45257 every 8 connection(s)
I20260926 12:30:08.211115 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 19038
I20260926 12:30:08.211241 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:30:08.211553 7564 external_mini_cluster.cc:1428] Setting key 5d915e216ac92843cf1fe1088379d998
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:08.230643 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.213171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:51251 (local address 127.7.99.62:33365)
0926 12:30:08.213288 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.213292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.214239 (+ 947us) server_negotiation.cc:407] Connection header received
0926 12:30:08.215449 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.215453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.215507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.215558 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:08.217251 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.218043 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.219720 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.220096 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.220197 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.226040 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:08.226044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:08.226047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:08.226092 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:08.228015 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.228715 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.228719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.228721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.228812 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.229249 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.229252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.229254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.229485 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:08.229576 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.230280 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.230445 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:30:08.232574 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.212019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33365 (local address 127.7.99.2:51251)
0926 12:30:08.214089 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.214145 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.215211 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.215955 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.215963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.216346 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:08.217004 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.217024 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.218215 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.218218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.219582 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.219593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.220412 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.220416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.220611 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.223955 (+ 3344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:08.223977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:08.225818 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:08.228212 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:08.228218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:08.228236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:08.228589 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:08.228926 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:08.228929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:08.228931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:08.229129 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:08.229647 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:08.229653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:08.229988 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.231126 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.231478 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1842,"thread_start_us":145,"threads_started":1}
I20260926 12:30:08.234175 19168 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33365
I20260926 12:30:08.234428 19168 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:08.235008 19168 heartbeater.cc:511] Master 127.7.99.62:33365 requested a full tablet report, sending...
I20260926 12:30:08.236238 18818 ts_manager.cc:194] Registered new tserver with Master: eb1664fee5bb4adfb9280ee8a4eb8150 (127.7.99.2:45257)
I20260926 12:30:08.236974 18818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:51251
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:30:08.277712 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:48747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
W20260926 12:30:08.448396 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:08.449120 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:08.449244 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:08.455374 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:08.455498 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:08.455633 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:30:08.463232 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.99.20:48747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:08.465247 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:08.466537 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:08.479681 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:08.480023 19173 server_base.cc:1034] running on GCE node
W20260926 12:30:08.483971 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:08.484190 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:08.484858 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:08.485651 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:08.486807 19173 hybrid_clock.cc:648] HybridClock initialized: now 1790425808486745 us; error 81 us; skew 500 ppm
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:08.490504 19173 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:30:08.491667 19173 webserver.cc:460] Webserver started at http://127.7.99.3:39155/ using document root <none> and password file <none>
I20260926 12:30:08.492501 19173 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:08.492630 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:08.492901 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:08.508126 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "d05cc9d4502c45adaa4c3c6428978e59"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "08068e13ac48a20b3e0c51bdd1398509"
server_key_iv: "a24e939ff706d60c91554823f3651837"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.508894 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "d05cc9d4502c45adaa4c3c6428978e59"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "08068e13ac48a20b3e0c51bdd1398509"
server_key_iv: "a24e939ff706d60c91554823f3651837"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.514667 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 12:30:08.532976 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.534723 19173 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 12:30:08.534852 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "d05cc9d4502c45adaa4c3c6428978e59"
format_stamp: "Formatted at 2026-09-26 12:30:08 on dist-test-slave-3jw2"
server_key: "08068e13ac48a20b3e0c51bdd1398509"
server_key_iv: "a24e939ff706d60c91554823f3651837"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:08.534976 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:08.566496 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:08.570180 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:08.570425 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:08.571134 19173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:08.572139 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:08.572273 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.572363 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:08.572420 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:08.584832 19173 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:37349
I20260926 12:30:08.586018 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:30:08.587447 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:37349 every 8 connection(s)
I20260926 12:30:08.590394 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 19173
I20260926 12:30:08.590515 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:30:08.590837 7564 external_mini_cluster.cc:1428] Setting key 222ca4398662882114267b97fb13af23
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425808, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:08.606196 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.590922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:47495 (local address 127.7.99.62:33365)
0926 12:30:08.591280 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.591285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.591304 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:30:08.593462 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.593465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.593515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.593561 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:08.595155 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.595936 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.597555 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.597880 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.597953 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.600914 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:08.600918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:08.600921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:08.600947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:08.602594 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.603172 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.603175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.603177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.603262 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.604929 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.604933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.604935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.605161 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:08.605244 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.605919 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.606042 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260926 12:30:08.608263 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.589684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:33365 (local address 127.7.99.3:47495)
0926 12:30:08.590258 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.590331 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.593230 (+ 2899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.593966 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.593974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.594297 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:08.595003 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.595021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.596080 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.596091 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.597424 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.597434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.598139 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.598142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.598320 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.599019 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:08.599039 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:08.600712 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:08.602728 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:08.602733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:08.602748 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:08.603058 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:08.603919 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:08.603922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:08.603924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:08.604817 (+ 893us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:08.605308 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:08.605314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:08.605647 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.606744 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.607084 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":136,"threads_started":1}
I20260926 12:30:08.609863 19302 heartbeater.cc:347] Connected to a master server at 127.7.99.62:33365
I20260926 12:30:08.610100 19302 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:08.610684 19302 heartbeater.cc:511] Master 127.7.99.62:33365 requested a full tablet report, sending...
I20260926 12:30:08.611740 18818 ts_manager.cc:194] Registered new tserver with Master: d05cc9d4502c45adaa4c3c6428978e59 (127.7.99.3:37349)
I20260926 12:30:08.612459 18818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:47495
I20260926 12:30:08.616197 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:30:08.634212 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.624647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.7.99.62:33365)
0926 12:30:08.624760 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.624764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.625307 (+ 543us) server_negotiation.cc:407] Connection header received
0926 12:30:08.625476 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.625479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.625523 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.625569 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:08.626502 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.627432 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.628728 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.629038 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.629109 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.630805 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:08.630808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:08.630811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:08.630836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:08.632463 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.632930 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.632933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.632935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.633014 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:08.633301 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:08.633305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:08.633307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:08.633499 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:08.633606 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.633947 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.634061 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:30:08.637876 18818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:30:08.642434 18818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:30:08.665223 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.653656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:37349 (local address 127.0.0.1:59714)
0926 12:30:08.656855 (+ 3199us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.656904 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.657023 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.659161 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.659167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.659192 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:08.659351 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.659359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.661443 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.661446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.662776 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.662786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.664730 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.664734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.664835 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.664873 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.664971 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.665079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3025,"thread_start_us":89,"threads_started":1}
I20260926 12:30:08.666038 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.654575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.7.99.3:37349)
0926 12:30:08.658097 (+ 3522us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.658113 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.658144 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:30:08.658604 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.658611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.658831 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.658986 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:08.659480 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.661325 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.662923 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.664410 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.664520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.664671 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.665803 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.665901 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3377,"thread_start_us":130,"threads_started":1}
I20260926 12:30:08.667646 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.655971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:41179 (local address 127.0.0.1:37468)
0926 12:30:08.658309 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.658322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.658405 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.662150 (+ 3745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.662153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.662169 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:08.662255 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.662261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.663521 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.663524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.664642 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.664654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.667280 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.667283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.667378 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.667399 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.667466 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.667526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2279,"thread_start_us":71,"threads_started":1}
I20260926 12:30:08.669466 19236 tablet_service.cc:1467] Processing CreateTablet for tablet ee7ed574f012453394ea720d6370f885 (DEFAULT_TABLE table=test-table [id=ce617ea8aa6b46aca7e1dca97b066e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:08.670590 19236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7ed574f012453394ea720d6370f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:08.678889 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.656368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.7.99.1:41179)
0926 12:30:08.661562 (+ 5194us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.661569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.661592 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:30:08.661661 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.661666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.661855 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.661998 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:08.662376 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.663399 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.666207 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.666958 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.667065 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.667220 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.678596 (+ 11376us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.678731 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5088,"thread_start_us":77,"threads_started":1}
I20260926 12:30:08.680886 19321 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Bootstrap starting.
I20260926 12:30:08.682049 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.656261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:45257 (local address 127.0.0.1:44232)
0926 12:30:08.661234 (+ 4973us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.661247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.661327 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.664457 (+ 3130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.664460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.664477 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:08.664584 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.664590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.665993 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.665996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.667149 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.667161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.681583 (+ 14422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.681587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.681698 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.681726 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.681810 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.681900 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4916,"thread_start_us":90,"threads_started":1}
I20260926 12:30:08.682543 18966 tablet_service.cc:1467] Processing CreateTablet for tablet ee7ed574f012453394ea720d6370f885 (DEFAULT_TABLE table=test-table [id=ce617ea8aa6b46aca7e1dca97b066e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:08.683161 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.656416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.7.99.2:45257)
0926 12:30:08.663796 (+ 7380us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.663812 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.663838 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:30:08.663913 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.663918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.664109 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.664325 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:08.664717 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.665859 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.680276 (+ 14417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.681214 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.681340 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.681489 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.682339 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.682439 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7265,"thread_start_us":67,"threads_started":1}
I20260926 12:30:08.683700 18966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7ed574f012453394ea720d6370f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:08.683912 19321 tablet_bootstrap.cc:654] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:08.685809 19321 log.cc:824] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:08.686048 19101 tablet_service.cc:1467] Processing CreateTablet for tablet ee7ed574f012453394ea720d6370f885 (DEFAULT_TABLE table=test-table [id=ce617ea8aa6b46aca7e1dca97b066e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:08.687076 19101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7ed574f012453394ea720d6370f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:08.698354 19321 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: No bootstrap required, opened a new log
I20260926 12:30:08.698685 19321 ts_tablet_manager.cc:1397] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 12:30:08.702781 19321 raft_consensus.cc:348] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.703269 19321 raft_consensus.cc:374] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:08.703387 19321 raft_consensus.cc:729] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05cc9d4502c45adaa4c3c6428978e59, State: Initialized, Role: FOLLOWER
I20260926 12:30:08.703972 19321 consensus_queue.cc:260] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.705179 19321 ts_tablet_manager.cc:1428] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:30:08.705610 19302 heartbeater.cc:503] Master 127.7.99.62:33365 was elected leader, sending a full tablet report...
I20260926 12:30:08.710464 19323 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Bootstrap starting.
I20260926 12:30:08.713071 19323 tablet_bootstrap.cc:654] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:08.715407 19324 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Bootstrap starting.
I20260926 12:30:08.717702 19323 log.cc:824] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:08.719485 19324 tablet_bootstrap.cc:654] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:08.721146 19324 log.cc:824] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:08.724586 19323 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: No bootstrap required, opened a new log
I20260926 12:30:08.724855 19323 ts_tablet_manager.cc:1397] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260926 12:30:08.729740 19323 raft_consensus.cc:348] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.730135 19323 raft_consensus.cc:374] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:08.730239 19323 raft_consensus.cc:729] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57a7f08f7914b589d90b4edcbbc1af5, State: Initialized, Role: FOLLOWER
I20260926 12:30:08.730845 19323 consensus_queue.cc:260] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.731998 19323 ts_tablet_manager.cc:1428] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:30:08.732442 19032 heartbeater.cc:503] Master 127.7.99.62:33365 was elected leader, sending a full tablet report...
I20260926 12:30:08.738900 19324 tablet_bootstrap.cc:492] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: No bootstrap required, opened a new log
I20260926 12:30:08.739158 19324 ts_tablet_manager.cc:1397] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20260926 12:30:08.743816 19324 raft_consensus.cc:348] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.744299 19324 raft_consensus.cc:374] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:08.744407 19324 raft_consensus.cc:729] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1664fee5bb4adfb9280ee8a4eb8150, State: Initialized, Role: FOLLOWER
I20260926 12:30:08.744984 19324 consensus_queue.cc:260] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.746054 19324 ts_tablet_manager.cc:1428] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:30:08.746465 19168 heartbeater.cc:503] Master 127.7.99.62:33365 was elected leader, sending a full tablet report...
Sep 26 12:30:08 dist-test-slave-3jw2 krb5kdc[18342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425807, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:08.838438 19328 raft_consensus.cc:482] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:30:08.838678 19328 raft_consensus.cc:504] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
W20260926 12:30:08.839277 19303 tablet.cc:2369] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:30:08.840114 19328 leader_election.cc:288] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d05cc9d4502c45adaa4c3c6428978e59 (127.7.99.3:37349), eb1664fee5bb4adfb9280ee8a4eb8150 (127.7.99.2:45257)
I20260926 12:30:08.848218 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.840628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:38877 (local address 127.7.99.3:37349)
0926 12:30:08.840751 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.840755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.843167 (+ 2412us) server_negotiation.cc:407] Connection header received
0926 12:30:08.843532 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.843536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.843593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.843648 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:08.844125 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.844956 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.846566 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.847139 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.847223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.847289 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.847982 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.848063 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 12:30:08.848706 19033 tablet.cc:2369] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:30:08.849484 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.840545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:37349 (local address 127.7.99.1:38877)
0926 12:30:08.843061 (+ 2516us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.843077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.843334 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.843818 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.843821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.843856 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:08.843995 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.844001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.845124 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.845128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.846423 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.846434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.847740 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.847744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.847836 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.847858 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.849248 (+ 1390us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.849332 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2444,"thread_start_us":108,"threads_started":1}
I20260926 12:30:08.850209 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7ed574f012453394ea720d6370f885" candidate_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05cc9d4502c45adaa4c3c6428978e59" is_pre_election: true
I20260926 12:30:08.850530 19240 raft_consensus.cc:2454] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57a7f08f7914b589d90b4edcbbc1af5 in term 0.
I20260926 12:30:08.851732 18920 leader_election.cc:302] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d05cc9d4502c45adaa4c3c6428978e59, e57a7f08f7914b589d90b4edcbbc1af5; no voters:
I20260926 12:30:08.852006 19328 raft_consensus.cc:2781] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:30:08.852070 19328 raft_consensus.cc:482] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:30:08.852110 19328 raft_consensus.cc:3037] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:08.853644 19328 raft_consensus.cc:504] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.854046 19328 leader_election.cc:288] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [CANDIDATE]: Term 1 election: Requested vote from peers d05cc9d4502c45adaa4c3c6428978e59 (127.7.99.3:37349), eb1664fee5bb4adfb9280ee8a4eb8150 (127.7.99.2:45257)
I20260926 12:30:08.854641 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7ed574f012453394ea720d6370f885" candidate_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05cc9d4502c45adaa4c3c6428978e59"
I20260926 12:30:08.854784 19240 raft_consensus.cc:3037] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:08.856144 19240 raft_consensus.cc:2454] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57a7f08f7914b589d90b4edcbbc1af5 in term 1.
I20260926 12:30:08.856604 18920 leader_election.cc:302] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d05cc9d4502c45adaa4c3c6428978e59, e57a7f08f7914b589d90b4edcbbc1af5; no voters:
I20260926 12:30:08.856844 19328 raft_consensus.cc:2781] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:30:08.857054 19328 raft_consensus.cc:686] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [term 1 LEADER]: Becoming Leader. State: Replica: e57a7f08f7914b589d90b4edcbbc1af5, State: Running, Role: LEADER
I20260926 12:30:08.857391 19328 consensus_queue.cc:237] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } }
I20260926 12:30:08.860836 18818 catalog_manager.cc:5495] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57a7f08f7914b589d90b4edcbbc1af5 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 41179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 } health_report { overall_health: UNKNOWN } } }
I20260926 12:30:08.865123 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.841602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:33953 (local address 127.7.99.2:45257)
0926 12:30:08.841731 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.841735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.850002 (+ 8267us) server_negotiation.cc:407] Connection header received
0926 12:30:08.850188 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.850191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.850242 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.850289 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:08.850682 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.851412 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.863397 (+ 11985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.864053 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.864180 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.864252 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.864912 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.864997 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:30:08.865588 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.840975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:45257 (local address 127.7.99.1:33953)
0926 12:30:08.849930 (+ 8955us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.849944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:08.850070 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:08.850451 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:08.850453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:08.850472 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:08.850548 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.850553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.862081 (+ 11528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.862084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.863225 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:08.863239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.864647 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.864650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:08.864753 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.864779 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:30:08.865382 (+ 603us) client_negotiation.cc:240] Negotiation successful
0926 12:30:08.865463 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8897,"thread_start_us":95,"threads_started":1}
I20260926 12:30:08.866446 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7ed574f012453394ea720d6370f885" candidate_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" is_pre_election: true
I20260926 12:30:08.866765 19106 raft_consensus.cc:2454] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57a7f08f7914b589d90b4edcbbc1af5 in term 0.
I20260926 12:30:08.867091 19119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7ed574f012453394ea720d6370f885" candidate_uuid: "e57a7f08f7914b589d90b4edcbbc1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150"
I20260926 12:30:08.867200 19119 raft_consensus.cc:3037] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:08.869094 19119 raft_consensus.cc:2454] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57a7f08f7914b589d90b4edcbbc1af5 in term 1.
I20260926 12:30:08.904337 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.899260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.7.99.1:41179)
0926 12:30:08.899406 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:08.899411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:08.899428 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:30:08.899519 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:08.899522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:08.899586 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:08.899697 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:30:08.900090 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.900912 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.902128 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:08.902446 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:08.902549 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:08.902990 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:30:08.903557 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:30:08.903674 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:08.904110 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 12:30:08.904191 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:30:08.912078 19119 raft_consensus.cc:1261] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 1 FOLLOWER]: Refusing update from remote peer e57a7f08f7914b589d90b4edcbbc1af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:30:08.913020 19334 consensus_queue.cc:1035] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1664fee5bb4adfb9280ee8a4eb8150" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:30:08.917600 19240 raft_consensus.cc:1261] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 1 FOLLOWER]: Refusing update from remote peer e57a7f08f7914b589d90b4edcbbc1af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:30:08.918517 19334 consensus_queue.cc:1035] T ee7ed574f012453394ea720d6370f885 P e57a7f08f7914b589d90b4edcbbc1af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d05cc9d4502c45adaa4c3c6428978e59" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:30:08.935170 19338 mvcc.cc:205] Tried to move back new op lower bound from 7333584113293332480 to 7333584113082527744. Current Snapshot: MvccSnapshot[applied={T|T < 7333584113293332480}]
I20260926 12:30:08.950634 18818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:30:08.950841 18818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:30:08.953464 18818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33dc28b55ffc49dbb6df8921bade5490: Sending DeleteTablet for 3 replicas of tablet ee7ed574f012453394ea720d6370f885
I20260926 12:30:08.954460 19236 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7ed574f012453394ea720d6370f885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:08 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:59714
I20260926 12:30:08.954677 18966 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7ed574f012453394ea720d6370f885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:08 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:37468
I20260926 12:30:08.955169 19346 tablet_replica.cc:330] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: stopping tablet replica
I20260926 12:30:08.955139 19101 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7ed574f012453394ea720d6370f885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:08 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:44232
I20260926 12:30:08.955622 19346 raft_consensus.cc:2229] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:30:08.956059 19346 raft_consensus.cc:2258] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:30:08.956151 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 18899
I20260926 12:30:08.959826 19346 ts_tablet_manager.cc:1905] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:30:08.962574 19346 ts_tablet_manager.cc:1918] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:30:08.962754 19346 log.cc:1196] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/ee7ed574f012453394ea720d6370f885
I20260926 12:30:08.963152 19346 ts_tablet_manager.cc:1939] T ee7ed574f012453394ea720d6370f885 P d05cc9d4502c45adaa4c3c6428978e59: Deleting consensus metadata
I20260926 12:30:08.964438 18803 catalog_manager.cc:4841] TS d05cc9d4502c45adaa4c3c6428978e59 (127.7.99.3:37349): tablet ee7ed574f012453394ea720d6370f885 (table test-table [id=ce617ea8aa6b46aca7e1dca97b066e2f]) successfully deleted
I20260926 12:30:08.964960 19348 tablet_replica.cc:330] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: stopping tablet replica
I20260926 12:30:08.965302 19348 raft_consensus.cc:2229] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 12:30:08.965585 18802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:41179 (error 108)
I20260926 12:30:08.965647 19348 raft_consensus.cc:2258] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:30:08.967070 19348 ts_tablet_manager.cc:1905] T ee7ed574f012453394ea720d6370f885 P eb1664fee5bb4adfb9280ee8a4eb8150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:30:08.967146 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.966832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:41179 (local address 127.0.0.1:37478)
0926 12:30:08.967000 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.967064 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 12:30:08.967351 18802 catalog_manager.cc:4584] TS e57a7f08f7914b589d90b4edcbbc1af5 (127.7.99.1:41179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee7ed574f012453394ea720d6370f885: Network error: Client connection negotiation failed: client connection to 127.7.99.1:41179: connect: Connection refused (error 111)
I20260926 12:30:08.967666 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 19038
I20260926 12:30:08.974328 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 19173
I20260926 12:30:08.974874 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:08.974578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:45257 (local address 127.0.0.1:44242)
0926 12:30:08.974747 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:08.974799 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 12:30:08.981803 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 18786
I20260926 12:30:08.987934 7564 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:30:09.332796 7564 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:30:09.332948 7564 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:30:09.678103 7564 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:30:09.678 UTC [18372] LOG: received smart shutdown request
2026-09-26 12:30:09.682 UTC [18372] LOG: background worker "logical replication launcher" (PID 18381) exited with exit code 1
2026-09-26 12:30:09.682 UTC [18376] LOG: shutting down
2026-09-26 12:30:09.845 UTC [18372] LOG: database system is shut down
2026-09-26T12:30:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46113 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:30:10 dist-test-slave-3jw2 krb5kdc[19358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:30:10 dist-test-slave-3jw2 krb5kdc[19358](info): set up 2 sockets
Sep 26 12:30:10 dist-test-slave-3jw2 krb5kdc[19358](info): commencing operation
krb5kdc: starting...
W20260926 12:30:11.004455 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:30:11 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:30:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangeradmin_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangeradmin_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangerlookup_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangerlookup_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
I20260926 12:30:11.166898 7564 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT6LLMM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/postgres -l logfile start
2026-09-26 12:30:13.633 UTC [19388] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:30:13.633 UTC [19388] LOG: listening on IPv4 address "127.7.99.20", port 50819
2026-09-26 12:30:13.637 UTC [19388] LOG: listening on Unix socket "/tmp/.s.PGSQL.50819"
2026-09-26 12:30:13.671 UTC [19391] LOG: database system was shut down at 2026-09-26 12:30:12 UTC
2026-09-26 12:30:13.674 UTC [19388] LOG: database system is ready to accept connections
I20260926 12:30:14.624321 7564 mini_postgres.cc:96] Postgres bound to 50819
2026-09-26 12:30:14.630 UTC [19399] FATAL: database "slave" does not exist
127.7.99.20:50819 - accepting connections
I20260926 12:30:14.631255 7564 mini_ranger.cc:161] Starting Ranger...
I20260926 12:30:14.646173 7564 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:30:15.098383 7564 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:30:15.098471 7564 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin
2026-09-26 12:30:15,366 [I] DB FLAVOR :POSTGRES
2026-09-26 12:30:15,366 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:30:15,367 [I] Checking connection..
2026-09-26 12:30:15,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:30:15,607 [I] Checking connection passed.
2026-09-26 12:30:15,607 [I] --------- Verifying version history table ---------
2026-09-26 12:30:15,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:30:15,866 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:30:15,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:30:16,114 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:30:16,114 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:30:16,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:30:16.338 UTC [19463] WARNING: there is no transaction in progress
2026-09-26 12:30:16,345 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:30:16,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:30:16,585 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:30:16,585 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:30:16,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:30:16,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:30:17,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3jw2.c.gcp-upstream.internal','N') ;"
2026-09-26 12:30:17,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:30:17,552 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:30:17,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:30:17,809 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:30:17,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:30:18,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:30:18,273 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:30:18,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:30:19.192 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.209 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.224 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.239 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.254 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.270 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.320 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.327 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.341 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.351 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.360 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.374 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.383 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.389 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.396 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:19.402 UTC [19571] WARNING: there is no transaction in progress
2026-09-26 12:30:20,026 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:30:20,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:30:20,251 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:30:20,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:30:20,484 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:30:20,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:30:20,738 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:30:20,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:30:20,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:30:21,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3jw2.c.gcp-upstream.internal';"
2026-09-26 12:30:21,428 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:30:21,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:30:21,677 [I] DB_PATCHES have already been applied
I20260926 12:30:21.683235 7564 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT6LLMM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:30:21.683296 7564 mini_ranger.cc:193] Using host: 127.7.99.20
I20260926 12:30:21.686818 7564 mini_ranger.cc:227] Ranger admin URL: http://127.7.99.20:50569
Sep 26, 2026 12:30:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:30:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:30:22.803622 7564 mini_ranger.cc:160] Time spent starting Ranger: real 8.172s user 0.000s sys 0.009s
Sep 26, 2026 12:30:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:30:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 12:30:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:30:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:30:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:30:44.440140 7564 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangerkms_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/rangerkms_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.99.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.99.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.99.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/HTTP_127.7.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:30:44 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425844, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:30:44.526860 7564 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:30:44.536600 7564 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:30:45.403507 7564 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:30:45.403573 7564 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms
I20260926 12:30:45.403585 7564 mini_ranger_kms.cc:227] Using postgres at 127.7.99.20:50819
2026-09-26 12:30:45,568 [I] DB FLAVOR :POSTGRES
2026-09-26 12:30:45,569 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:30:45,569 [I] Checking connection
2026-09-26 12:30:45,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:30:45,799 [I] connection success
2026-09-26 12:30:45,799 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:30:45,799 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:30:45,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:30:46,037 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:30:46,037 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:30:46,038 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:30:46,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.99.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:30:46,299 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:30:46.752586 7564 mini_ranger_kms.cc:310] Created kms service
I20260926 12:30:46.927039 7564 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:30:46.970759 7564 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:30:46 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425844, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:47.348629 7564 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:30:47.348783 7564 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT6LLMM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT6LLMM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:30:47.348802 7564 mini_ranger_kms.cc:240] Using host: 127.7.99.20
I20260926 12:30:47.352500 7564 mini_ranger_kms.cc:276] Ranger KMS PID: 19751
I20260926 12:30:47.352577 7564 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.99.20:49813
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:30:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:30:49.628196 7564 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.101s user 0.007s sys 0.142s
I20260926 12:30:49.628338 7564 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:30:49.628436 7564 mini_ranger_kms.cc:401] 127.7.99.20:49813/kms/v1/keys
Sep 26 12:30:49 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790425849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:30:50 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
Sep 26 12:30:50 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.99.20@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
Sep 26 12:30:52 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:30:52.799782 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44887
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44887
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:52.926776 19802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:52.927106 19802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:52.927214 19802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:52.931351 19802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:52.931440 19802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:52.931478 19802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:52.931535 19802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:52.931571 19802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:52.937027 19802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.99.20:49813/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44887
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:52.938059 19802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:52.939060 19802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:52.947541 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:52.947587 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:52.947659 19802 server_base.cc:1034] running on GCE node
W20260926 12:30:52.947916 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:52.948527 19802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:52.949545 19802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:52.950726 19802 hybrid_clock.cc:648] HybridClock initialized: now 1790425852950705 us; error 44 us; skew 500 ppm
Sep 26 12:30:52 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425852, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:52.962114 19802 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:30:52.962968 19802 webserver.cc:460] Webserver started at http://127.7.99.62:44943/ using document root <none> and password file <none>
I20260926 12:30:52.963505 19802 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:52.963587 19802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:52.963809 19802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:52 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425852, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:53.016688 19802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "4a3c648e082e499f9f84400ab69471c9"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "75efec6763525396ba20ef08b824ca0b"
server_key_iv: "f1ea49c352d253bad720b260e89a125d"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:53.017429 19802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "4a3c648e082e499f9f84400ab69471c9"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "75efec6763525396ba20ef08b824ca0b"
server_key_iv: "f1ea49c352d253bad720b260e89a125d"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:53.022284 19802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 12:30:53.036109 19817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:53.037607 19802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:30:53.037756 19802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "4a3c648e082e499f9f84400ab69471c9"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "75efec6763525396ba20ef08b824ca0b"
server_key_iv: "f1ea49c352d253bad720b260e89a125d"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:53.037891 19802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:53.048718 19802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:53.051585 19802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:53.051761 19802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:53.059406 19802 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44887
I20260926 12:30:53.059415 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44887 every 8 connection(s)
I20260926 12:30:53.060497 19802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:30:53.063843 19870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:53.067427 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 19802
I20260926 12:30:53.067556 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:30:53.067822 7564 external_mini_cluster.cc:1428] Setting key 5fc5c64d497879bc900ac522920ee021
I20260926 12:30:53.071007 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Bootstrap starting.
I20260926 12:30:53.073827 19870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:53.074689 19870 log.cc:824] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:30:53 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:53.076900 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: No bootstrap required, opened a new log
I20260926 12:30:53.081293 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:53.069240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.7.99.62:44887)
0926 12:30:53.069622 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:53.069637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:53.069670 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:30:53.070276 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:53.070292 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:53.070614 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:53.070954 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:53.071744 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:53.072732 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:53.073655 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:53.074185 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:53.074357 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:53.077020 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:53.077026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:53.077038 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:53.077068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:53.078671 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:53.079086 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:53.079095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:53.079099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:53.079205 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:53.079477 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:53.079480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:53.079481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:53.079828 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:53.080014 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:53.080295 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 12:30:53.080605 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":111,"threads_started":1}
I20260926 12:30:53.081143 19870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:53.081390 19870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:53.081444 19870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3c648e082e499f9f84400ab69471c9, State: Initialized, Role: FOLLOWER
I20260926 12:30:53.081923 19870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:53.082108 19870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:30:53.082217 19870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:30:53.082315 19870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:53.083467 19870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:53.083876 19870 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3c648e082e499f9f84400ab69471c9; no voters:
I20260926 12:30:53.084233 19870 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:30:53.084355 19875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:30:53.084857 19875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3c648e082e499f9f84400ab69471c9, State: Running, Role: LEADER
I20260926 12:30:53.085232 19875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:53.085467 19870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:30:53.086333 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3c648e082e499f9f84400ab69471c9. Latest consensus state: current_term: 1 leader_uuid: "4a3c648e082e499f9f84400ab69471c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } } }
I20260926 12:30:53.086529 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:53.086719 19877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a3c648e082e499f9f84400ab69471c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } } }
I20260926 12:30:53.086858 19877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:53.086958 19884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:30:53.089999 19884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:30:53.090624 19870 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:30:53.102692 19884 catalog_manager.cc:1347] Generated new cluster ID: f083cfcb2ff543b2a54f9eecc4663e10
I20260926 12:30:53.102772 19884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:30:53.126434 19884 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:30:53.127861 19884 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:30:53.135294 19884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Generated new TSK 0
I20260926 12:30:53.136219 19884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:30:53.807911 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:53.928962 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:53.929250 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:53.929328 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:53.933586 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:53.933663 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:53.933748 19915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:30:53.940387 19915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:53.941867 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:53.943169 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:53.954105 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:53.954110 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:53.954290 19915 server_base.cc:1034] running on GCE node
W20260926 12:30:53.954465 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:53.954911 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:53.955554 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:53.956725 19915 hybrid_clock.cc:648] HybridClock initialized: now 1790425853956714 us; error 32 us; skew 500 ppm
Sep 26 12:30:53 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425853, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:53.959635 19915 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:30:53.960623 19915 webserver.cc:460] Webserver started at http://127.7.99.1:34837/ using document root <none> and password file <none>
I20260926 12:30:53.961169 19915 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:53.961261 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:53.961484 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:53 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425853, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:53.977226 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "24265355e3a84f8fb4fcad692e5ee770"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "43c280199ce0d194e6f86996be61b7e3"
server_key_iv: "127f132d3978b8e29e895d1a6fd9732f"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:53.978025 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "24265355e3a84f8fb4fcad692e5ee770"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "43c280199ce0d194e6f86996be61b7e3"
server_key_iv: "127f132d3978b8e29e895d1a6fd9732f"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:53.982990 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260926 12:30:54.007258 19932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.009263 19915 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 12:30:54.009423 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "24265355e3a84f8fb4fcad692e5ee770"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "43c280199ce0d194e6f86996be61b7e3"
server_key_iv: "127f132d3978b8e29e895d1a6fd9732f"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.009573 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:54.047204 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:54.051083 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:54.051272 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:54.051973 19915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:54.053002 19915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:54.053063 19915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.053117 19915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:54.053138 19915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.063657 19915 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:45561
I20260926 12:30:54.066049 20045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:45561 every 8 connection(s)
I20260926 12:30:54.066831 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:30:54.075570 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 19915
I20260926 12:30:54.075693 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:30:54.076020 7564 external_mini_cluster.cc:1428] Setting key 69e8aa33b6cafbbeccd243bc944b9dc9
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425853, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:30:54.095455 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.069438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:33827 (local address 127.7.99.62:44887)
0926 12:30:54.069750 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:54.069759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:54.070525 (+ 766us) server_negotiation.cc:407] Connection header received
0926 12:30:54.071504 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:54.071507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:54.071573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:54.071688 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:54.073004 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.073873 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.075589 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.075982 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.076066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.078977 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:54.078981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:54.078984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:54.079010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:54.080911 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.093344 (+ 12433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.093349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.093351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.093468 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.093955 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.093958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.093960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.094229 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:54.094326 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:54.095100 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 12:30:54.095260 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260926 12:30:54.097430 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.069931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.1:33827)
0926 12:30:54.070378 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:54.070428 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:54.071295 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:54.071860 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:54.071867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:54.072132 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:54.072835 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.072855 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.074057 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.074061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.075449 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.075460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.076294 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.076298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.076491 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.077117 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:54.077139 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:54.078792 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:54.090642 (+ 11850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.090651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.090672 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.093165 (+ 2493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.093610 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.093613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.093615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.093835 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.094396 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:54.094404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:54.094789 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 12:30:54.095786 (+ 997us) client_negotiation.cc:240] Negotiation successful
0926 12:30:54.096251 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":140,"threads_started":1}
I20260926 12:30:54.099242 20046 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:54.099502 20046 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:54.100209 20046 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:54.102181 19834 ts_manager.cc:194] Registered new tserver with Master: 24265355e3a84f8fb4fcad692e5ee770 (127.7.99.1:45561)
I20260926 12:30:54.103899 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:33827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:30:54.112917 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 12:30:54.275111 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:54.275570 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:54.275687 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:54.282020 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:54.282114 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:54.282208 20054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:30:54.290292 20054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:54.291867 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:54.293339 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:54.306085 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:54.306088 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:54.306439 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:54.306201 20054 server_base.cc:1034] running on GCE node
I20260926 12:30:54.306926 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:54.307739 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:54.308949 20054 hybrid_clock.cc:648] HybridClock initialized: now 1790425854308931 us; error 40 us; skew 500 ppm
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:54.312799 20054 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:30:54.313917 20054 webserver.cc:460] Webserver started at http://127.7.99.2:42219/ using document root <none> and password file <none>
I20260926 12:30:54.314687 20054 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:54.314776 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:54.315100 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:54.332847 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "10f5ec05384e491195c4ffbffe250759"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "98024739f902a5c5ad49c9987561a088"
server_key_iv: "337458920d5712d3d8b4642f80a78d3b"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.333855 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "10f5ec05384e491195c4ffbffe250759"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "98024739f902a5c5ad49c9987561a088"
server_key_iv: "337458920d5712d3d8b4642f80a78d3b"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.345933 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260926 12:30:54.380692 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.382759 20054 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260926 12:30:54.383033 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "10f5ec05384e491195c4ffbffe250759"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "98024739f902a5c5ad49c9987561a088"
server_key_iv: "337458920d5712d3d8b4642f80a78d3b"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.383276 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:54.407809 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:54.411893 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:54.412209 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:54.413038 20054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:54.414152 20054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:54.414258 20054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.414361 20054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:54.414450 20054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.429514 20054 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:35801
I20260926 12:30:54.430860 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:30:54.432054 20183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:35801 every 8 connection(s)
I20260926 12:30:54.433092 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20054
I20260926 12:30:54.433229 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:30:54.433599 7564 external_mini_cluster.cc:1428] Setting key b2286d13d3288fef8763e3b25f4b8aa2
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:54.469269 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.436317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:36247 (local address 127.7.99.62:44887)
0926 12:30:54.437468 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:54.437473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:54.441727 (+ 4254us) server_negotiation.cc:407] Connection header received
0926 12:30:54.447502 (+ 5775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:54.447506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:54.447593 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:54.447658 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:54.449530 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.450525 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.457949 (+ 7424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.458373 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.458471 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.464776 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:54.464781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:54.464784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:54.464819 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:54.466685 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.467372 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.467376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.467378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.467476 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.467910 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.467913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.467915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.468150 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:54.468272 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:54.468944 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 12:30:54.469085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032}
I20260926 12:30:54.470454 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.435182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.2:36247)
0926 12:30:54.441538 (+ 6356us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:54.441614 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:54.443463 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:54.448149 (+ 4686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:54.448190 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:54.448592 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:54.449334 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.449355 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.450699 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.450703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.452360 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.452372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.458723 (+ 6351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.458727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.458975 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.459767 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:54.459790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:54.464537 (+ 4747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:54.466838 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.466844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.466865 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.467228 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.467587 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.467590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.467593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.467787 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.468279 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:54.468286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:54.468649 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 12:30:54.468869 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 12:30:54.469237 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6099,"thread_start_us":144,"threads_started":1}
I20260926 12:30:54.471851 20184 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:54.472142 20184 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:54.472761 20184 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:54.473938 19834 ts_manager.cc:194] Registered new tserver with Master: 10f5ec05384e491195c4ffbffe250759 (127.7.99.2:35801)
I20260926 12:30:54.474747 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:36247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:30:54.492373 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:54.682358 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:54.682762 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:54.682848 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
W20260926 12:30:54.689698 20189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:54.689777 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:54.689908 20189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:30:54.698019 20189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:54.699497 20189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:54.700896 20189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:54.712587 20195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:54.713002 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:54.713577 20194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:54.718042 20189 server_base.cc:1034] running on GCE node
I20260926 12:30:54.718550 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:54.719398 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:54.722731 20189 hybrid_clock.cc:648] HybridClock initialized: now 1790425854722650 us; error 88 us; skew 500 ppm
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:54.726859 20189 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:30:54.728238 20189 webserver.cc:460] Webserver started at http://127.7.99.3:35345/ using document root <none> and password file <none>
I20260926 12:30:54.729031 20189 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:54.729128 20189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:54.729365 20189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:54.764362 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "6c3aa733508146038e1d798eb72baa8c"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "22f1f73382956a8910095d03d35bca14"
server_key_iv: "abd9de1c775f2f91c2595e5651340af5"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.765285 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "6c3aa733508146038e1d798eb72baa8c"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "22f1f73382956a8910095d03d35bca14"
server_key_iv: "abd9de1c775f2f91c2595e5651340af5"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.779836 20189 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260926 12:30:54.799188 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.800825 20189 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260926 12:30:54.800984 20189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "6c3aa733508146038e1d798eb72baa8c"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "22f1f73382956a8910095d03d35bca14"
server_key_iv: "abd9de1c775f2f91c2595e5651340af5"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:54.801154 20189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:54.819018 20189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:54.822283 20189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:54.822482 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:54.823069 20189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:54.823889 20189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:54.823961 20189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.824038 20189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:54.824070 20189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:54.833658 20189 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:36771
I20260926 12:30:54.834848 20189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:30:54.835603 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20189
I20260926 12:30:54.835713 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:30:54.836017 7564 external_mini_cluster.cc:1428] Setting key 08dbdd19a8bf40a33a237729f971e03e
I20260926 12:30:54.838932 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:36771 every 8 connection(s)
Sep 26 12:30:54 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:54.855691 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.841517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:41895 (local address 127.7.99.62:44887)
0926 12:30:54.841666 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:54.841671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:54.842795 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 12:30:54.843947 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:54.843950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:54.844005 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:54.844105 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:54.845565 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.846396 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.848203 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.848541 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.848648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.851526 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:54.851530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:54.851533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:54.851562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:54.853318 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.853916 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.853920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.853922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.854019 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:54.854419 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:54.854423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:54.854424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:54.854654 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:54.854749 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:54.855407 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 12:30:54.855529 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:30:54.856746 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:54.841450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.3:41895)
0926 12:30:54.842642 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:54.842696 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:54.843708 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:54.844332 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:54.844342 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:54.844700 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:54.845399 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.845418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.846544 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.846551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.848043 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:54.848055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:54.848719 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:54.848723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:54.848925 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:54.849668 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:54.849689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:54.851350 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:54.853447 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.853454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.853470 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.853810 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.854120 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:54.854123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:54.854126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:54.854304 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:54.854774 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:54.854781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:54.855124 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 12:30:54.855338 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 12:30:54.855668 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":159,"threads_started":1}
I20260926 12:30:54.858008 20318 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:54.858269 20318 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:54.858819 20318 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:54.859965 19834 ts_manager.cc:194] Registered new tserver with Master: 6c3aa733508146038e1d798eb72baa8c (127.7.99.3:36771)
I20260926 12:30:54.860749 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:41895
I20260926 12:30:54.862505 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:30:54.863444 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 19915
I20260926 12:30:54.871389 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20054
I20260926 12:30:54.878633 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20189
I20260926 12:30:54.883430 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 19802
I20260926 12:30:54.890432 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44887
--webserver_interface=127.7.99.62
--webserver_port=44943
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44887
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:55.000137 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:55.000401 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:55.000439 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:55.004328 20321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:55.004377 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:55.004395 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:55.004411 20321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:55.004426 20321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:55.009773 20321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44887
--ranger_config_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:55.010757 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:55.011741 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:55.020114 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.020114 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:55.020248 20321 server_base.cc:1034] running on GCE node
W20260926 12:30:55.020190 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:55.020987 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:55.022159 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:55.023342 20321 hybrid_clock.cc:648] HybridClock initialized: now 1790425855023328 us; error 26 us; skew 500 ppm
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:55.026232 20321 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:30:55.027225 20321 webserver.cc:460] Webserver started at http://127.7.99.62:44943/ using document root <none> and password file <none>
I20260926 12:30:55.027805 20321 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:55.027905 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:55.092597 20321 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:30:55.094941 20337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.096333 20321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:30:55.096478 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "4a3c648e082e499f9f84400ab69471c9"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "75efec6763525396ba20ef08b824ca0b"
server_key_iv: "f1ea49c352d253bad720b260e89a125d"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:55.097014 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:55.110303 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:55.112757 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:55.112944 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:55.119928 20321 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44887
I20260926 12:30:55.119935 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44887 every 8 connection(s)
I20260926 12:30:55.121008 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:30:55.125388 20390 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:30:55.127003 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Bootstrap starting.
I20260926 12:30:55.127866 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20321
I20260926 12:30:55.128369 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:45561
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:44887
--webserver_port=34837
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:30:55.148737 20390 log.cc:824] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:55.157069 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:30:55.157495 20390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Bootstrap complete.
I20260926 12:30:55.162912 20390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:55.163419 20390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3c648e082e499f9f84400ab69471c9, State: Initialized, Role: FOLLOWER
I20260926 12:30:55.163992 20390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:55.164135 20390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:30:55.164258 20390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:30:55.164356 20390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:30:55.166772 20390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:55.167132 20390 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3c648e082e499f9f84400ab69471c9; no voters:
I20260926 12:30:55.167497 20390 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:30:55.167655 20394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:30:55.167917 20394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [term 2 LEADER]: Becoming Leader. State: Replica: 4a3c648e082e499f9f84400ab69471c9, State: Running, Role: LEADER
I20260926 12:30:55.168293 20394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } }
I20260926 12:30:55.168787 20390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:30:55.169631 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3c648e082e499f9f84400ab69471c9. Latest consensus state: current_term: 2 leader_uuid: "4a3c648e082e499f9f84400ab69471c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } } }
I20260926 12:30:55.169742 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:55.170481 20403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:30:55.170387 20395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a3c648e082e499f9f84400ab69471c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3c648e082e499f9f84400ab69471c9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44887 } } }
I20260926 12:30:55.170902 20395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:30:55.173712 20403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:30:55.174230 20390 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:30:55.174369 20403 catalog_manager.cc:1259] Loaded cluster ID: f083cfcb2ff543b2a54f9eecc4663e10
I20260926 12:30:55.174417 20403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:30:55.179441 20403 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:30:55.181106 20403 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Loaded TSK: 0
I20260926 12:30:55.182593 20403 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:30:55.284050 20392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:55.284521 20392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:55.284646 20392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:55.290616 20392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:55.290695 20392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:55.290782 20392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:30:55.298491 20392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:45561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=34837
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:55.299824 20392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:55.301195 20392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:55.313560 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.313787 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.314082 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:55.314622 20392 server_base.cc:1034] running on GCE node
I20260926 12:30:55.315146 20392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:55.315919 20392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:55.317093 20392 hybrid_clock.cc:648] HybridClock initialized: now 1790425855317072 us; error 40 us; skew 500 ppm
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:55.320716 20392 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:30:55.322212 20392 webserver.cc:460] Webserver started at http://127.7.99.1:34837/ using document root <none> and password file <none>
I20260926 12:30:55.322962 20392 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:55.323115 20392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:55.346391 20392 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260926 12:30:55.349620 20439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.351166 20392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:30:55.351282 20392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "24265355e3a84f8fb4fcad692e5ee770"
format_stamp: "Formatted at 2026-09-26 12:30:53 on dist-test-slave-3jw2"
server_key: "43c280199ce0d194e6f86996be61b7e3"
server_key_iv: "127f132d3978b8e29e895d1a6fd9732f"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:55.352012 20392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:55.380328 20392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:55.384189 20392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:55.384451 20392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:55.385155 20392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:55.386284 20392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:55.386392 20392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.386520 20392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:55.386606 20392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.397962 20392 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:45561
I20260926 12:30:55.397981 20552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:45561 every 8 connection(s)
I20260926 12:30:55.399075 20392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:30:55.406471 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20392
I20260926 12:30:55.406996 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:35801
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:44887
--webserver_port=42219
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:55.421618 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.401664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.1:40559)
0926 12:30:55.404123 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:55.404209 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:55.405240 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:55.406842 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:55.406850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:55.407246 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:55.407778 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.407792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.409668 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.409672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.411217 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.411232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.412462 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.412469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.412705 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.413439 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:55.413466 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:55.415118 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:55.417818 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.417827 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.417843 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.418203 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.418630 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.418634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.418636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.418801 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.419545 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:55.419551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:55.419916 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 12:30:55.420109 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 12:30:55.420476 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2228,"thread_start_us":133,"threads_started":1}
I20260926 12:30:55.424037 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.403087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:40559 (local address 127.7.99.62:44887)
0926 12:30:55.405715 (+ 2628us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:55.405736 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:55.405779 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:30:55.406054 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:55.406078 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:55.406434 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:55.406781 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:55.407967 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.409503 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.411385 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.412267 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.413163 (+ 896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.415357 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:55.415365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:55.415383 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:55.415414 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:55.417674 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.418351 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.418357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.418363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.418498 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.418912 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.418914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.418916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.419413 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:55.422668 (+ 3255us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:55.422819 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 12:30:55.423114 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474,"thread_start_us":107,"threads_started":1}
I20260926 12:30:55.426806 20553 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:55.427048 20553 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.427624 20553 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
W20260926 12:30:55.429877 20553 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44887 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:30:55.429993 20553 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.430172 20553 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:55.431077 20553 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.431234 20553 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
W20260926 12:30:55.432032 20553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:30:55.539762 20557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:55.540025 20557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:55.540340 20557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:55.545748 20557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:55.545840 20557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:55.545928 20557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:30:55.551970 20557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:35801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=42219
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:55.553264 20557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:55.554332 20557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:55.564214 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.564337 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.564218 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:55.564947 20557 server_base.cc:1034] running on GCE node
I20260926 12:30:55.565375 20557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:55.566056 20557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:55.567297 20557 hybrid_clock.cc:648] HybridClock initialized: now 1790425855567278 us; error 40 us; skew 500 ppm
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:55.571448 20557 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:30:55.572775 20557 webserver.cc:460] Webserver started at http://127.7.99.2:42219/ using document root <none> and password file <none>
I20260926 12:30:55.573493 20557 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:55.573606 20557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:55.595309 20557 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:30:55.603590 20573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.605049 20557 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260926 12:30:55.605155 20557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "10f5ec05384e491195c4ffbffe250759"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "98024739f902a5c5ad49c9987561a088"
server_key_iv: "337458920d5712d3d8b4642f80a78d3b"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:55.606024 20557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:55.619416 20557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:55.622292 20557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:55.622486 20557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:55.623214 20557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:55.624130 20557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:55.624228 20557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.624318 20557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:55.624353 20557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.636477 20557 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:35801
I20260926 12:30:55.638825 20557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:30:55.640664 20686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:35801 every 8 connection(s)
I20260926 12:30:55.645432 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20557
I20260926 12:30:55.645927 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.99.20:49813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:36771
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:44887
--webserver_port=35345
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:55.659040 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.642719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:44795 (local address 127.7.99.62:44887)
0926 12:30:55.642845 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:55.642849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:55.642864 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:30:55.643323 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:55.643327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:55.643385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:55.643531 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:55.644655 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.645262 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.647028 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.647430 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.647538 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.652847 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:55.652854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:55.652859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:55.652894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:55.654928 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.656622 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.656628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.656631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.656729 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.657271 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.657277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.657280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.657536 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:55.657653 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:55.658612 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 12:30:55.658790 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:30:55.660924 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.641499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.2:44795)
0926 12:30:55.642103 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:55.642145 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:55.643124 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:55.643674 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:55.643681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:55.643978 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:55.644513 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.644527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.645416 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.645421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.646878 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.646892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.647742 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.647746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.647966 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.648708 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:55.648732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:55.652630 (+ 3898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:55.656034 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.656042 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.656061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.656481 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.656858 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.656863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.656865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.657145 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.657749 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:55.657757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:55.658252 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 12:30:55.659515 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0926 12:30:55.659853 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":170,"threads_started":1}
I20260926 12:30:55.662770 20687 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:55.663029 20687 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.663560 20687 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
W20260926 12:30:55.665126 20687 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44887 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:30:55.665241 20687 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.665385 20687 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:55.666142 20687 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.666297 20687 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
W20260926 12:30:55.667106 20687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:30:55.785017 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:55.785352 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:55.785497 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:55.790961 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:55.791072 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:55.791193 20691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:30:55.797868 20691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.99.20:49813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:36771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=35345
--tserver_master_addrs=127.7.99.62:44887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:55.798933 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:55.799998 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:55.811211 20697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.811606 20696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:55.812206 20699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:55.813567 20691 server_base.cc:1034] running on GCE node
I20260926 12:30:55.814033 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:55.814831 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:55.818130 20691 hybrid_clock.cc:648] HybridClock initialized: now 1790425855818087 us; error 51 us; skew 500 ppm
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:55.821228 20691 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:30:55.822184 20691 webserver.cc:460] Webserver started at http://127.7.99.3:35345/ using document root <none> and password file <none>
I20260926 12:30:55.822794 20691 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:55.822898 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:55.843591 20691 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 12:30:55.847834 20707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.849618 20691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 12:30:55.849838 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "6c3aa733508146038e1d798eb72baa8c"
format_stamp: "Formatted at 2026-09-26 12:30:54 on dist-test-slave-3jw2"
server_key: "22f1f73382956a8910095d03d35bca14"
server_key_iv: "abd9de1c775f2f91c2595e5651340af5"
server_key_version: "kuduclusterkey@0"
I20260926 12:30:55.850739 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:55.868327 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:55.872634 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:55.872851 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:55.873672 20691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:30:55.874910 20691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:30:55.875038 20691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.875155 20691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:30:55.875201 20691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:55.887676 20691 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:36771
I20260926 12:30:55.887799 20820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:36771 every 8 connection(s)
I20260926 12:30:55.889192 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:30:55.894837 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20691
Sep 26 12:30:55 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:55.911641 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.893605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:55943 (local address 127.7.99.62:44887)
0926 12:30:55.893755 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:55.893761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:55.896703 (+ 2942us) server_negotiation.cc:407] Connection header received
0926 12:30:55.898153 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:55.898157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:55.898243 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:55.898311 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:55.900321 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.901303 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.903364 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.903728 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.903814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.907057 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:55.907062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:55.907066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:55.907096 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:55.908883 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.909589 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.909594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.909596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.909686 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.910114 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.910118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.910120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.910382 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:55.910473 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:55.911276 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 12:30:55.911426 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:30:55.912896 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.892399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44887 (local address 127.7.99.3:55943)
0926 12:30:55.896529 (+ 4130us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:55.896591 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:55.897831 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:55.898775 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:55.898786 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:55.899189 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:55.900114 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.900137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.901660 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.901666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.903211 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:55.903223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.904054 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.904058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:55.904288 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.904995 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:55.905020 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:55.906822 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:55.909049 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.909056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.909074 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.909459 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.909803 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:55.909806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:55.909809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:55.909998 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:55.910614 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:55.910623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:55.910964 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 12:30:55.911174 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 12:30:55.911532 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3881,"thread_start_us":142,"threads_started":1}
I20260926 12:30:55.914428 20821 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44887
I20260926 12:30:55.914707 20821 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.915331 20821 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:55.916760 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:55.895650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.7.99.62:44887)
0926 12:30:55.901181 (+ 5531us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:55.901186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:55.901199 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:30:55.901233 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:55.901236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:55.901275 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:55.901357 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:55.902357 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.903191 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.906809 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:55.907169 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:55.907260 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:55.912443 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:55.912446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:55.912448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:55.912468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:55.914245 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.914744 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.914748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.914749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.914851 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:55.915187 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:55.915193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:55.915198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:55.915423 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:55.915642 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:55.916326 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 12:30:55.916579 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5457,"thread_start_us":100,"threads_started":1}
W20260926 12:30:55.916961 20821 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44887 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:30:55.917083 20821 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.917232 20821 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:55.918133 20821 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:55.918291 20821 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
W20260926 12:30:55.919062 20821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 12:30:56 dist-test-slave-3jw2 krb5kdc[19358](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790425856, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:56.432278 20553 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:56.432569 20553 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:56.433784 20353 ts_manager.cc:194] Registered new tserver with Master: 24265355e3a84f8fb4fcad692e5ee770 (127.7.99.1:45561)
I20260926 12:30:56.435529 20353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:40559
Sep 26 12:30:56 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425856, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:56.667315 20687 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:56.667614 20687 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
I20260926 12:30:56.668332 20353 ts_manager.cc:194] Registered new tserver with Master: 10f5ec05384e491195c4ffbffe250759 (127.7.99.2:35801)
I20260926 12:30:56.668919 20353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:44795
I20260926 12:30:56.919262 20821 heartbeater.cc:464] Registering TS with master...
I20260926 12:30:56.919529 20821 heartbeater.cc:511] Master 127.7.99.62:44887 requested a full tablet report, sending...
Sep 26 12:30:56 dist-test-slave-3jw2 krb5kdc[19358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425856, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for HTTP/127.7.99.20@KRBTEST.COM
I20260926 12:30:56.925791 20353 ts_manager.cc:194] Registered new tserver with Master: 6c3aa733508146038e1d798eb72baa8c (127.7.99.3:36771)
I20260926 12:30:56.926646 20353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:55943
I20260926 12:30:56.938995 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:30:56.953743 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.942520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.7.99.62:44887)
0926 12:30:56.943167 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:56.943171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:56.943521 (+ 350us) server_negotiation.cc:407] Connection header received
0926 12:30:56.943565 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:56.943568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:56.943619 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:56.943694 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:56.945474 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.946362 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.947557 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.947913 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.948046 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:56.949770 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:56.949774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:56.949777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:56.949801 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:56.951529 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:56.952010 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:56.952014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:56.952016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:56.952108 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:56.952475 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:56.952479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:56.952481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:56.952686 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:56.952805 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:56.953444 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 12:30:56.953565 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":80,"threads_started":1}
I20260926 12:30:56.958017 20353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:30:56.963160 20353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:30:56.999009 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.983281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45561 (local address 127.0.0.1:51378)
0926 12:30:56.983937 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:56.983986 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:56.988267 (+ 4281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:56.992968 (+ 4701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:56.992980 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:56.993005 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:56.993148 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:56.993155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.994550 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.994554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:56.996924 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:56.996933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.998452 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.998455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:56.998580 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:56.998617 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 12:30:56.998725 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:30:56.998845 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 12:30:56.999699 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.984299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.7.99.1:45561)
0926 12:30:56.991954 (+ 7655us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:56.991963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:56.991991 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:30:56.992078 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:56.992084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:56.992602 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:56.992768 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:56.993296 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.994410 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.997056 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.997970 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.998227 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:56.998401 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:56.999439 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 12:30:56.999560 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7513,"thread_start_us":118,"threads_started":1}
I20260926 12:30:57.003252 20469 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c (DEFAULT_TABLE table=test-table [id=2c2ab6b81e934cc5be10b8e5ef5ac16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:57.004618 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0e134b2f6240a1b0b8c2237fae637c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:57.016209 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.985148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:35801 (local address 127.0.0.1:48474)
0926 12:30:56.993888 (+ 8740us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:56.993903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:56.993980 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:57.010483 (+ 16503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:57.010487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:57.010512 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:57.010643 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.010651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.012307 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.012311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.013416 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.013429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.015743 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.015746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.015843 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.015868 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:30:57.015932 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:30:57.016026 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8677,"thread_start_us":99,"threads_started":1}
I20260926 12:30:57.016839 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.985214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.7.99.2:35801)
0926 12:30:57.001629 (+ 16415us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:57.001636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:57.001662 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:30:57.001742 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:57.001748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:57.001960 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:57.002116 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:57.010818 (+ 8702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.012092 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.014630 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.015399 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.015478 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.015636 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:57.016580 (+ 944us) server_negotiation.cc:299] Negotiation successful
0926 12:30:57.016688 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16295,"thread_start_us":63,"threads_started":1}
I20260926 12:30:57.020989 20602 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c (DEFAULT_TABLE table=test-table [id=2c2ab6b81e934cc5be10b8e5ef5ac16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:57.022184 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0e134b2f6240a1b0b8c2237fae637c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:57.017704 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.983607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36771 (local address 127.0.0.1:45468)
0926 12:30:56.993030 (+ 9423us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:56.993046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:56.993134 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:56.994065 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:56.994068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:56.994080 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:56.994197 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:56.994203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.995641 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.995644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:56.999618 (+ 3974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:56.999628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.017256 (+ 17628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.017259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.017357 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.017395 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:30:57.017465 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:30:57.017558 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9360,"mutex_wait_us":2714,"thread_start_us":85,"threads_started":1}
I20260926 12:30:57.028858 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:56.984700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.7.99.3:36771)
0926 12:30:56.992384 (+ 7684us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:56.992391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:56.993237 (+ 846us) server_negotiation.cc:407] Connection header received
0926 12:30:56.993313 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:56.993318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:56.993500 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:56.993647 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:56.994322 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:56.995496 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:56.999754 (+ 4258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.014238 (+ 14484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.014361 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.014526 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:57.028568 (+ 14042us) server_negotiation.cc:299] Negotiation successful
0926 12:30:57.028707 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7564,"thread_start_us":114,"threads_started":1}
I20260926 12:30:57.032109 20746 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c (DEFAULT_TABLE table=test-table [id=2c2ab6b81e934cc5be10b8e5ef5ac16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:30:57.033283 20746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0e134b2f6240a1b0b8c2237fae637c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:57.038854 20857 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Bootstrap starting.
I20260926 12:30:57.040027 20856 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Bootstrap starting.
I20260926 12:30:57.043107 20857 tablet_bootstrap.cc:654] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:57.043860 20857 log.cc:824] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:57.044898 20856 tablet_bootstrap.cc:654] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:57.045809 20856 log.cc:824] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:57.046543 20858 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Bootstrap starting.
I20260926 12:30:57.047062 20857 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: No bootstrap required, opened a new log
I20260926 12:30:57.047250 20857 ts_tablet_manager.cc:1397] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260926 12:30:57.048734 20858 tablet_bootstrap.cc:654] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Neither blocks nor log segments found. Creating new log.
I20260926 12:30:57.049485 20858 log.cc:824] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:30:57.051048 20857 raft_consensus.cc:348] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.051242 20857 raft_consensus.cc:374] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:57.051280 20857 raft_consensus.cc:729] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f5ec05384e491195c4ffbffe250759, State: Initialized, Role: FOLLOWER
I20260926 12:30:57.051687 20857 consensus_queue.cc:260] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.052287 20687 heartbeater.cc:503] Master 127.7.99.62:44887 was elected leader, sending a full tablet report...
I20260926 12:30:57.053812 20857 ts_tablet_manager.cc:1428] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260926 12:30:57.056357 20856 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: No bootstrap required, opened a new log
I20260926 12:30:57.056356 20858 tablet_bootstrap.cc:492] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: No bootstrap required, opened a new log
I20260926 12:30:57.056560 20858 ts_tablet_manager.cc:1397] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 12:30:57.056550 20856 ts_tablet_manager.cc:1397] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260926 12:30:57.062047 20858 raft_consensus.cc:348] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.062414 20858 raft_consensus.cc:374] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:57.062511 20858 raft_consensus.cc:729] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3aa733508146038e1d798eb72baa8c, State: Initialized, Role: FOLLOWER
I20260926 12:30:57.063064 20858 consensus_queue.cc:260] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.062983 20856 raft_consensus.cc:348] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.063263 20856 raft_consensus.cc:374] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:30:57.063345 20856 raft_consensus.cc:729] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24265355e3a84f8fb4fcad692e5ee770, State: Initialized, Role: FOLLOWER
I20260926 12:30:57.063817 20856 consensus_queue.cc:260] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.064275 20858 ts_tablet_manager.cc:1428] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 12:30:57.064656 20821 heartbeater.cc:503] Master 127.7.99.62:44887 was elected leader, sending a full tablet report...
I20260926 12:30:57.064873 20856 ts_tablet_manager.cc:1428] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 12:30:57.065171 20553 heartbeater.cc:503] Master 127.7.99.62:44887 was elected leader, sending a full tablet report...
I20260926 12:30:57.090304 20864 raft_consensus.cc:482] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:30:57.090523 20864 raft_consensus.cc:504] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.091861 20864 leader_election.cc:288] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f5ec05384e491195c4ffbffe250759 (127.7.99.2:35801), 6c3aa733508146038e1d798eb72baa8c (127.7.99.3:36771)
I20260926 12:30:57.108917 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.092547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:56167 (local address 127.7.99.2:35801)
0926 12:30:57.092700 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:57.092705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:57.094617 (+ 1912us) server_negotiation.cc:407] Connection header received
0926 12:30:57.095009 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:57.095013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:57.095070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:57.095123 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:57.095872 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.096681 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.107498 (+ 10817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.108176 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.108265 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.108334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:57.108700 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 12:30:57.108782 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:30:57.109754 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.092802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:35801 (local address 127.7.99.1:56167)
0926 12:30:57.094522 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:57.094537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:57.094697 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:57.095528 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:57.095532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:57.095553 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:57.095728 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.095736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.102686 (+ 6950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.102689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.104151 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.104184 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.108428 (+ 4244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.108432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.108546 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.108570 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:30:57.109506 (+ 936us) client_negotiation.cc:240] Negotiation successful
0926 12:30:57.109587 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1652,"thread_start_us":127,"threads_started":1}
I20260926 12:30:57.111783 20641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0e134b2f6240a1b0b8c2237fae637c" candidate_uuid: "24265355e3a84f8fb4fcad692e5ee770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f5ec05384e491195c4ffbffe250759" is_pre_election: true
I20260926 12:30:57.112129 20641 raft_consensus.cc:2454] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24265355e3a84f8fb4fcad692e5ee770 in term 0.
I20260926 12:30:57.112771 20440 leader_election.cc:302] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f5ec05384e491195c4ffbffe250759, 24265355e3a84f8fb4fcad692e5ee770; no voters:
I20260926 12:30:57.113087 20864 raft_consensus.cc:2781] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:30:57.113162 20864 raft_consensus.cc:482] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:30:57.113204 20864 raft_consensus.cc:3037] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:57.114540 20864 raft_consensus.cc:504] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.114897 20864 leader_election.cc:288] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [CANDIDATE]: Term 1 election: Requested vote from peers 10f5ec05384e491195c4ffbffe250759 (127.7.99.2:35801), 6c3aa733508146038e1d798eb72baa8c (127.7.99.3:36771)
I20260926 12:30:57.115470 20641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0e134b2f6240a1b0b8c2237fae637c" candidate_uuid: "24265355e3a84f8fb4fcad692e5ee770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f5ec05384e491195c4ffbffe250759"
I20260926 12:30:57.115584 20641 raft_consensus.cc:3037] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:57.117065 20641 raft_consensus.cc:2454] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24265355e3a84f8fb4fcad692e5ee770 in term 1.
I20260926 12:30:57.117422 20440 leader_election.cc:302] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f5ec05384e491195c4ffbffe250759, 24265355e3a84f8fb4fcad692e5ee770; no voters:
I20260926 12:30:57.117627 20864 raft_consensus.cc:2781] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:30:57.117821 20864 raft_consensus.cc:686] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [term 1 LEADER]: Becoming Leader. State: Replica: 24265355e3a84f8fb4fcad692e5ee770, State: Running, Role: LEADER
I20260926 12:30:57.118180 20864 consensus_queue.cc:237] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } }
I20260926 12:30:57.121529 20354 catalog_manager.cc:5495] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24265355e3a84f8fb4fcad692e5ee770 (127.7.99.1). New cstate: current_term: 1 leader_uuid: "24265355e3a84f8fb4fcad692e5ee770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24265355e3a84f8fb4fcad692e5ee770" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 } health_report { overall_health: UNKNOWN } } }
I20260926 12:30:57.123075 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.093553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:49471 (local address 127.7.99.3:36771)
0926 12:30:57.093659 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:57.093663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:57.104517 (+ 10854us) server_negotiation.cc:407] Connection header received
0926 12:30:57.104718 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:57.104722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:57.104777 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:57.104827 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:30:57.105232 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.106009 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.110041 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.110708 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.110789 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.110847 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:57.122830 (+ 11983us) server_negotiation.cc:299] Negotiation successful
0926 12:30:57.122944 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:30:57.123839 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.093246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36771 (local address 127.7.99.1:49471)
0926 12:30:57.104429 (+ 11183us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:57.104444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:57.104598 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:57.104954 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:57.104956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:57.104974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:30:57.105104 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.105110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.106151 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.106154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.107349 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:57.107360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.111017 (+ 3657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.111021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:57.111119 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.111139 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:30:57.123605 (+ 12466us) client_negotiation.cc:240] Negotiation successful
0926 12:30:57.123693 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11109,"thread_start_us":76,"threads_started":1}
I20260926 12:30:57.124734 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0e134b2f6240a1b0b8c2237fae637c" candidate_uuid: "24265355e3a84f8fb4fcad692e5ee770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3aa733508146038e1d798eb72baa8c" is_pre_election: true
I20260926 12:30:57.125022 20757 raft_consensus.cc:2454] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24265355e3a84f8fb4fcad692e5ee770 in term 0.
I20260926 12:30:57.125388 20766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0e134b2f6240a1b0b8c2237fae637c" candidate_uuid: "24265355e3a84f8fb4fcad692e5ee770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3aa733508146038e1d798eb72baa8c"
I20260926 12:30:57.125511 20766 raft_consensus.cc:3037] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:30:57.126811 20766 raft_consensus.cc:2454] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24265355e3a84f8fb4fcad692e5ee770 in term 1.
W20260926 12:30:57.141769 20822 tablet.cc:2369] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:30:57.144081 20688 tablet.cc:2369] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:30:57.145260 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.138821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.7.99.1:45561)
0926 12:30:57.138957 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:57.138962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:57.138977 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:30:57.139076 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:57.139079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:57.139151 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:57.139271 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:30:57.139568 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.140365 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.142422 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:57.142777 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:57.142870 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:57.143257 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:30:57.143778 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:30:57.143903 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:57.145058 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0926 12:30:57.145120 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:30:57.152518 20766 raft_consensus.cc:1261] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 1 FOLLOWER]: Refusing update from remote peer 24265355e3a84f8fb4fcad692e5ee770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 12:30:57.153018 20554 tablet.cc:2369] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:30:57.153839 20867 consensus_queue.cc:1035] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3aa733508146038e1d798eb72baa8c" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:30:57.160923 20641 raft_consensus.cc:1261] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 1 FOLLOWER]: Refusing update from remote peer 24265355e3a84f8fb4fcad692e5ee770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:30:57.161640 20867 consensus_queue.cc:1035] T 2a0e134b2f6240a1b0b8c2237fae637c P 24265355e3a84f8fb4fcad692e5ee770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f5ec05384e491195c4ffbffe250759" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:30:57.174297 20870 mvcc.cc:205] Tried to move back new op lower bound from 7333584310887911424 to 7333584310759223296. Current Snapshot: MvccSnapshot[applied={T|T < 7333584310887911424}]
I20260926 12:30:57.177944 20873 mvcc.cc:205] Tried to move back new op lower bound from 7333584310887911424 to 7333584310759223296. Current Snapshot: MvccSnapshot[applied={T|T < 7333584310887911424}]
I20260926 12:30:57.185783 20354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:30:57.185967 20354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:30:57.192119 20354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a3c648e082e499f9f84400ab69471c9: Sending DeleteTablet for 3 replicas of tablet 2a0e134b2f6240a1b0b8c2237fae637c
I20260926 12:30:57.193656 20469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:57 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:51378
I20260926 12:30:57.194957 20746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:57 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:45468
I20260926 12:30:57.196396 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20392
I20260926 12:30:57.193121 20602 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0e134b2f6240a1b0b8c2237fae637c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:30:57 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48474
I20260926 12:30:57.202907 20871 mvcc.cc:205] Tried to move back new op lower bound from 7333584310887911424 to 7333584310759223296. Current Snapshot: MvccSnapshot[applied={T|T < 7333584310887911424}]
I20260926 12:30:57.204419 20880 tablet_replica.cc:330] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: stopping tablet replica
I20260926 12:30:57.204854 20880 raft_consensus.cc:2229] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:30:57.205219 20880 raft_consensus.cc:2258] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:30:57.208078 20879 tablet_replica.cc:330] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: stopping tablet replica
I20260926 12:30:57.208392 20879 raft_consensus.cc:2229] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:30:57.208671 20879 raft_consensus.cc:2258] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:30:57.208926 20880 ts_tablet_manager.cc:1905] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:30:57.209957 20879 ts_tablet_manager.cc:1905] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:30:57.211817 20880 ts_tablet_manager.cc:1918] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:30:57.211916 20880 log.cc:1196] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/2a0e134b2f6240a1b0b8c2237fae637c
I20260926 12:30:57.212296 20880 ts_tablet_manager.cc:1939] T 2a0e134b2f6240a1b0b8c2237fae637c P 10f5ec05384e491195c4ffbffe250759: Deleting consensus metadata
I20260926 12:30:57.213052 20879 ts_tablet_manager.cc:1918] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:30:57.213220 20879 log.cc:1196] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/2a0e134b2f6240a1b0b8c2237fae637c
I20260926 12:30:57.213609 20879 ts_tablet_manager.cc:1939] T 2a0e134b2f6240a1b0b8c2237fae637c P 6c3aa733508146038e1d798eb72baa8c: Deleting consensus metadata
I20260926 12:30:57.213598 20338 catalog_manager.cc:4841] TS 10f5ec05384e491195c4ffbffe250759 (127.7.99.2:35801): tablet 2a0e134b2f6240a1b0b8c2237fae637c (table test-table [id=2c2ab6b81e934cc5be10b8e5ef5ac16b]) successfully deleted
I20260926 12:30:57.214828 20341 catalog_manager.cc:4841] TS 6c3aa733508146038e1d798eb72baa8c (127.7.99.3:36771): tablet 2a0e134b2f6240a1b0b8c2237fae637c (table test-table [id=2c2ab6b81e934cc5be10b8e5ef5ac16b]) successfully deleted
W20260926 12:30:57.220340 20339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:45561 (error 108)
I20260926 12:30:57.221652 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20557
I20260926 12:30:57.221992 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:57.221671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:45561 (local address 127.0.0.1:51390)
0926 12:30:57.221854 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:57.221914 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 12:30:57.222134 20339 catalog_manager.cc:4584] TS 24265355e3a84f8fb4fcad692e5ee770 (127.7.99.1:45561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a0e134b2f6240a1b0b8c2237fae637c: Network error: Client connection negotiation failed: client connection to 127.7.99.1:45561: connect: Connection refused (error 111)
I20260926 12:30:57.229054 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20691
I20260926 12:30:57.236665 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20321
I20260926 12:30:57.243328 7564 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:30:57.578441 7564 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:30:57.578555 7564 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:30:57.915778 7564 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:30:57.915 UTC [19388] LOG: received smart shutdown request
2026-09-26 12:30:57.920 UTC [19388] LOG: background worker "logical replication launcher" (PID 19397) exited with exit code 1
2026-09-26 12:30:57.920 UTC [19392] LOG: shutting down
2026-09-26 12:30:58.087 UTC [19388] LOG: database system is shut down
2026-09-26T12:30:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48247 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:30:58 dist-test-slave-3jw2 krb5kdc[20891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:30:58 dist-test-slave-3jw2 krb5kdc[20891](info): set up 2 sockets
Sep 26 12:30:58 dist-test-slave-3jw2 krb5kdc[20891](info): commencing operation
krb5kdc: starting...
W20260926 12:30:59.248512 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:30:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:30:59.371007 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:37373
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:30:59.477449 20904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:59.477751 20904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:59.477823 20904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:59.481799 20904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:59.481880 20904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:59.481920 20904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:59.481956 20904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:59.481992 20904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:59.487063 20904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:37373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:59.488123 20904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:59.489166 20904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:59.497750 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.497750 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.497795 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:59.497915 20904 server_base.cc:1034] running on GCE node
I20260926 12:30:59.498620 20904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:59.499697 20904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:59.500878 20904 hybrid_clock.cc:648] HybridClock initialized: now 1790425859500866 us; error 26 us; skew 500 ppm
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:59.503566 20904 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:30:59.504460 20904 webserver.cc:460] Webserver started at http://127.7.99.62:35759/ using document root <none> and password file <none>
I20260926 12:30:59.504962 20904 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:59.505017 20904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:59.505178 20904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:30:59.507084 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "6a0de7eefd48092fea54d7c4150d744a"
server_key_iv: "480edc4c49232420c254e3b0cb344797"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.507582 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "6a0de7eefd48092fea54d7c4150d744a"
server_key_iv: "480edc4c49232420c254e3b0cb344797"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.511037 20904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:30:59.513350 20919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:59.514503 20904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:30:59.514635 20904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "6a0de7eefd48092fea54d7c4150d744a"
server_key_iv: "480edc4c49232420c254e3b0cb344797"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.514737 20904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:59.526139 20904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:59.528859 20904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:59.529073 20904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:59.535940 20904 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:37373
I20260926 12:30:59.535967 20971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:37373 every 8 connection(s)
I20260926 12:30:59.536931 20904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:30:59.539647 20972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:59.543380 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:30:59.546842 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20904
I20260926 12:30:59.546973 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:30:59.547283 7564 external_mini_cluster.cc:1428] Setting key 4027cdc4d7622305c07efdee3f275e60
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.61@KRBTEST.COM" created.
I20260926 12:30:59.559887 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.545577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37373 (local address 127.0.0.1:50796)
0926 12:30:59.546040 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.546098 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.547158 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.547885 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.547894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.548433 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.548833 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.548859 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.550523 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.550529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.551830 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.551844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.552562 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.552569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.552741 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.553348 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.553378 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.555023 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.557084 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.557092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.557102 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.557320 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.557675 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.557678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.557680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.557817 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.558474 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.558484 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.558657 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.558824 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.559102 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"spinlock_wait_cycles":32000,"thread_start_us":166,"threads_started":1}
I20260926 12:30:59.559906 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.545578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.7.99.62:37373)
0926 12:30:59.545990 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.546004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.546215 (+ 211us) server_negotiation.cc:407] Connection header received
0926 12:30:59.547491 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.547495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.547673 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.547827 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.549025 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.550396 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.551964 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.552394 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.552508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.555174 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.555179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.555189 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.555208 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.556942 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.557436 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.557449 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.557450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.557548 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.557908 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.557914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.557919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.558336 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.558482 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.558926 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.559240 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":267,"thread_start_us":144,"threads_started":1}
I20260926 12:30:59.562359 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:30:59.563064 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.562839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35456)
0926 12:30:59.562948 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.563010 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:30:59.563351 20922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
I20260926 12:30:59.564312 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.563953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35472)
0926 12:30:59.564131 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.564259 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 12:30:59.564509 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:37467: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
I20260926 12:30:59.575263 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:37467
--webserver_interface=127.7.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:30:59.613871 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } attempt: 1
I20260926 12:30:59.614779 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.614467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35476)
0926 12:30:59.614634 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.614701 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 12:30:59.615567 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.615351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35486)
0926 12:30:59.615459 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.615509 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:30:59.615733 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:37467: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
I20260926 12:30:59.683900 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } attempt: 2
I20260926 12:30:59.684758 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.684450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35496)
0926 12:30:59.684615 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.684684 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 12:30:59.685546 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.685321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35500)
0926 12:30:59.685428 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.685481 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:30:59.685720 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:37467: Network error: Client connection negotiation failed: client connection to 127.7.99.61:37467: connect: Connection refused (error 111)
W20260926 12:30:59.686998 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:59.687248 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:59.687346 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:30:59.691391 20979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:59.691439 20979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:59.691459 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:59.691476 20979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:59.691491 20979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:59.696892 20979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:59.697887 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:59.698866 20979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:30:59.707814 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.707823 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.707823 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:59.708249 20979 server_base.cc:1034] running on GCE node
I20260926 12:30:59.708803 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:59.710114 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:59.711308 20979 hybrid_clock.cc:648] HybridClock initialized: now 1790425859711279 us; error 45 us; skew 500 ppm
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:59.714046 20979 init.cc:377] Logged in from keytab as kudu/127.7.99.61@KRBTEST.COM (short username kudu)
I20260926 12:30:59.714883 20979 webserver.cc:460] Webserver started at http://127.7.99.61:33617/ using document root <none> and password file <none>
I20260926 12:30:59.715422 20979 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:59.715480 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:59.715703 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:30:59.717686 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data/instance:
uuid: "36e64a3faf9247f2bcd793094dc3c6a7"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "ca1df4b6e9cdbeb10ba39b7d8f311329"
server_key_iv: "2e6a9d3a64f50c77e19df2a882db08e4"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.718262 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal/instance:
uuid: "36e64a3faf9247f2bcd793094dc3c6a7"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "ca1df4b6e9cdbeb10ba39b7d8f311329"
server_key_iv: "2e6a9d3a64f50c77e19df2a882db08e4"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.721822 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:30:59.724224 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:59.725335 20979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:30:59.725457 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "36e64a3faf9247f2bcd793094dc3c6a7"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "ca1df4b6e9cdbeb10ba39b7d8f311329"
server_key_iv: "2e6a9d3a64f50c77e19df2a882db08e4"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.725559 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:59.745117 20979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:59.748387 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:59.748605 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:59.755303 20979 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.61:37467
I20260926 12:30:59.755362 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.61:37467 every 8 connection(s)
I20260926 12:30:59.756273 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
I20260926 12:30:59.758874 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:59.761399 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 20979
I20260926 12:30:59.761536 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-1/wal/instance
I20260926 12:30:59.761818 7564 external_mini_cluster.cc:1428] Setting key e037de9cc3e7949b2189b157a51b3903
I20260926 12:30:59.762538 21047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.99.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.60@KRBTEST.COM" created.
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:59.776674 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.764683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.7.99.62:37373)
0926 12:30:59.764833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.764838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.765608 (+ 770us) server_negotiation.cc:407] Connection header received
0926 12:30:59.766519 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.766527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.766603 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.766713 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.767781 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.768682 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.769812 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.770044 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.770146 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.772849 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.772856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.772863 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.772897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.774429 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.775041 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.775043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.775045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.775126 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.775519 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.775523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.775526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.775715 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.775789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.776442 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.776542 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:30:59.776865 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } attempt: 3
I20260926 12:30:59.777864 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.765036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37373 (local address 127.0.0.1:50804)
0926 12:30:59.765451 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.765499 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.766310 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.766859 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.766865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.767186 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.767628 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.767640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.768806 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.768809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.769702 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.769712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.770185 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.770188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.770318 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.770904 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.770918 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.772691 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.774553 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.774562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.774579 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.774938 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.775215 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.775221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.775226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.775411 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.775825 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.775833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.776156 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.776403 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.776772 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260926 12:30:59.779297 21047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
I20260926 12:30:59.787873 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:42319
--webserver_interface=127.7.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:30:59.788324 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.777293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35514)
0926 12:30:59.777461 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.777475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.777554 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.778210 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.778213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.778400 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.778964 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.778974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.780444 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.780447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.781173 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.781183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.781762 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.781766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.781868 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.782380 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.782400 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.783654 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.785916 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.785920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.785922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.786206 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.787134 (+ 928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.787140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.787143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.787241 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.787802 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.787805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.787896 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.788000 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.788119 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 12:30:59.788640 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.779816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35528)
0926 12:30:59.780270 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.780283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.780358 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.780609 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.780614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.780747 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.780832 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.780837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.781695 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.781701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.782407 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.782416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.782868 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.782871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.782948 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.783410 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.783425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.784721 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.786893 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.786896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.786897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.787178 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.787472 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.787475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.787477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.787559 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.787970 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.787976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.788076 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.788221 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.788365 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
I20260926 12:30:59.788662 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.779853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.7.99.61:37467)
0926 12:30:59.780094 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.780101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.780428 (+ 327us) server_negotiation.cc:407] Connection header received
0926 12:30:59.780452 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.780455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.780488 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.780572 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.780938 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.781593 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.782536 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.782766 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.782870 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.785275 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.785282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.785288 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.785314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.786793 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.787284 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.787289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.787294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.787369 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.787646 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.787651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.787652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.787866 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.787966 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.788322 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.788473 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260926 12:30:59.788899 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.777387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.7.99.61:37467)
0926 12:30:59.777765 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.777773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.777791 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:30:59.777823 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.777826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.778041 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.778822 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.779089 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.780342 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.781292 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.781611 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.781762 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.783768 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.783776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.783786 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.783807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.785729 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.786924 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.786931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.786935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.787023 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.787346 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.787350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.787351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.787683 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.787824 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.788648 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.788757 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260926 12:30:59.790495 20972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:30:59.790634 21047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:30:59.791239 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.791010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32852)
0926 12:30:59.791132 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.791192 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 12:30:59.791352 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.790930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32850)
0926 12:30:59.791210 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.791270 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260926 12:30:59.791417 20995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
I20260926 12:30:59.792181 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.791855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32862)
0926 12:30:59.792041 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.792100 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 12:30:59.792343 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.792065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32878)
0926 12:30:59.792216 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.792272 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:30:59.792335 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
W20260926 12:30:59.792505 21047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
I20260926 12:30:59.823503 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 1
I20260926 12:30:59.824354 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.824035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32890)
0926 12:30:59.824186 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.824257 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 12:30:59.825086 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.824876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32896)
0926 12:30:59.824976 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.825018 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:30:59.825228 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
I20260926 12:30:59.841804 21047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 1
I20260926 12:30:59.842615 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.842296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32908)
0926 12:30:59.842462 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.842536 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:30:59.843354 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.843137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32912)
0926 12:30:59.843240 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.843290 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:30:59.843493 21047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
W20260926 12:30:59.899066 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:30:59.899367 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:30:59.899446 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:30:59.900369 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 2
I20260926 12:30:59.901161 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.900876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32922)
0926 12:30:59.901010 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.901087 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 12:30:59.901796 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.901598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32936)
0926 12:30:59.901696 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.901740 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:30:59.901914 20972 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
W20260926 12:30:59.903863 21054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:30:59.903925 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:30:59.903959 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:30:59.903988 21054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:30:59.904011 21054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:30:59.909232 21054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:37373,127.7.99.61:37467,127.7.99.60:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:30:59.910195 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:30:59.911130 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:30:59.911660 21047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 2
I20260926 12:30:59.912396 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.912123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32944)
0926 12:30:59.912282 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.912332 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 12:30:59.913079 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.912855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32958)
0926 12:30:59.912970 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.913016 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:30:59.913196 21047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:42319: Network error: Client connection negotiation failed: client connection to 127.7.99.60:42319: connect: Connection refused (error 111)
W20260926 12:30:59.919677 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.919667 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:30:59.919703 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:30:59.919925 21054 server_base.cc:1034] running on GCE node
I20260926 12:30:59.920576 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:30:59.921622 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:30:59.922816 21054 hybrid_clock.cc:648] HybridClock initialized: now 1790425859922805 us; error 25 us; skew 500 ppm
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:30:59.925582 21054 init.cc:377] Logged in from keytab as kudu/127.7.99.60@KRBTEST.COM (short username kudu)
I20260926 12:30:59.926421 21054 webserver.cc:460] Webserver started at http://127.7.99.60:35287/ using document root <none> and password file <none>
I20260926 12:30:59.926978 21054 fs_manager.cc:359] Metadata directory not provided
I20260926 12:30:59.927066 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:30:59.927297 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:30:59.929430 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data/instance:
uuid: "93cf51d728b44e93954e4ab7ebea7de5"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "f9d2ea13a07b4a7c416fe911cfa54e6b"
server_key_iv: "e02ce132d731c3da4080ee4369ec46d6"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.930047 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal/instance:
uuid: "93cf51d728b44e93954e4ab7ebea7de5"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "f9d2ea13a07b4a7c416fe911cfa54e6b"
server_key_iv: "e02ce132d731c3da4080ee4369ec46d6"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.933732 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:30:59.935969 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:30:59.937199 21054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:30:59.937322 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "93cf51d728b44e93954e4ab7ebea7de5"
format_stamp: "Formatted at 2026-09-26 12:30:59 on dist-test-slave-3jw2"
server_key: "f9d2ea13a07b4a7c416fe911cfa54e6b"
server_key_iv: "e02ce132d731c3da4080ee4369ec46d6"
server_key_version: "encryptionkey@0"
I20260926 12:30:59.937423 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:30:59.955310 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:30:59.958220 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:30:59.958418 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:30:59.965085 21054 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.60:42319
I20260926 12:30:59.965112 21122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.60:42319 every 8 connection(s)
I20260926 12:30:59.965988 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
I20260926 12:30:59.968636 21123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:30:59.972206 21123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:30:59.975363 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21054
I20260926 12:30:59.975508 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425700812969-7564-0/minicluster-data/master-2/wal/instance
I20260926 12:30:59.975768 7564 external_mini_cluster.cc:1428] Setting key d3f8c0398a5160566b45c33be58f6441
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:30:59.986057 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.974128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.7.99.62:37373)
0926 12:30:59.974280 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.974285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.975010 (+ 725us) server_negotiation.cc:407] Connection header received
0926 12:30:59.976020 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.976023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.976082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.976215 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.977543 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.978053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.979058 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.979282 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.979397 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.982141 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.982144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.982146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.982182 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.983736 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.984335 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.984341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.984343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.984433 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.984762 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.984768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.984771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.985024 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.985100 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.985765 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.985888 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:30:59.986897 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.976440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.7.99.60:42319)
0926 12:30:59.976835 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.976849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.976874 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:30:59.976916 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.976921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.977090 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.977249 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.977979 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.978803 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.979614 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.980102 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.980235 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.982859 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.982865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.982875 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.982895 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.984890 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.985349 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.985353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.985354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.985424 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.985699 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.985701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.985704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.985967 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.986193 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.986385 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.986617 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":301,"spinlock_wait_cycles":1280,"thread_start_us":161,"threads_started":1}
I20260926 12:30:59.986907 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.974434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:37373 (local address 127.0.0.1:50818)
0926 12:30:59.974863 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.974905 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.975780 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.976376 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.976386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.976854 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.977400 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.977416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.978162 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.978168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.978933 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.978942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.979439 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.979443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.979616 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.980293 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.980312 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.981972 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.983837 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.983848 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.983861 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.984239 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.984534 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.984536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.984537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.984666 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.985126 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.985133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.985446 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.985691 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.985955 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":62,"spinlock_wait_cycles":2304,"thread_start_us":112,"threads_started":1}
I20260926 12:30:59.988250 21123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:30:59.997342 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.988774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:37467 (local address 127.0.0.1:35540)
0926 12:30:59.989032 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.989047 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.989128 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.989493 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.989495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.989625 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.989706 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.989710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.990689 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.990692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.991325 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.991334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.991895 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.991898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:30:59.992013 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.992522 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:30:59.992540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:30:59.993739 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:30:59.995771 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.995773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.995775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.995982 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.996342 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:30:59.996345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:30:59.996346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:30:59.996443 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:30:59.996903 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:30:59.996905 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:30:59.996977 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 12:30:59.997057 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:30:59.997181 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260926 12:30:59.997426 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.988893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.7.99.61:37467)
0926 12:30:59.989041 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.989045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.989086 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:30:59.989263 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.989266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.989324 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.989381 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.989812 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.990582 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.991434 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.991793 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.991870 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.993870 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.993874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.993877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.993902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.995664 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.996111 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.996114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.996115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.996240 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.996536 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.996540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.996541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.996808 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.996904 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.997138 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.997258 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:30:59.997666 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.989238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.7.99.62:37373)
0926 12:30:59.989340 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.989346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.989363 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:30:59.989480 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.989483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.989520 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.989610 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.990316 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.990886 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.991694 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:30:59.991941 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:30:59.992038 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:30:59.994422 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:30:59.994427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:30:59.994433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:30:59.994463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:30:59.995969 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.996467 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.996469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.996470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.996571 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:30:59.996830 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:30:59.996836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:30:59.996840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:30:59.997078 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:30:59.997297 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 12:30:59.997414 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 12:30:59.997526 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:30:59.998658 21123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:31:00.001092 20972 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 3
Sep 26 12:30:59 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:31:00.005146 21047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } attempt: 3
Sep 26 12:31:00 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
I20260926 12:31:00.007467 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.999134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32972)
0926 12:30:59.999302 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 12:30:59.999312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:30:59.999369 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:30:59.999578 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:30:59.999580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:30:59.999698 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:30:59.999762 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:30:59.999766 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.000781 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.000783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.001307 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:00.001316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.001940 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.001943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.002018 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.002514 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:00.002541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:00.003946 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:00.005875 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.005879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.005882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.006136 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.006429 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.006432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.006433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.006513 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.006937 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:00.006942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:00.007032 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 12:31:00.007155 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 12:31:00.007283 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
Sep 26 12:31:00 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:31:00.007673 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.999184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.7.99.60:42319)
0926 12:30:59.999292 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.999296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.999350 (+ 54us) server_negotiation.cc:407] Connection header received
0926 12:30:59.999427 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.999429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.999474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.999562 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.999871 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.000685 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.001419 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.001761 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.001846 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.004071 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.004075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.004080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.004104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.005760 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.006245 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.006249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.006250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.006330 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.006606 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.006609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.006611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.006804 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.006898 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.007398 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.007494 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:31:00.011387 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:30:59.998863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.7.99.61:37467)
0926 12:30:59.998984 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:30:59.998989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:30:59.999004 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:30:59.999055 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:30:59.999059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:30:59.999101 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:30:59.999176 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:30:59.999919 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.000686 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.001619 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.001930 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.001996 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.006696 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.006699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.006702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.006723 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.009145 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.009626 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.009629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.009631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.009707 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.009976 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.009978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.009980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.010210 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.010976 (+ 766us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.011068 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.011203 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:31:00.011521 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.001465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32982)
0926 12:31:00.001575 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:00.001590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:00.001674 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:00.002264 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:00.002268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:00.002454 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:00.002533 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:00.002538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.004028 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.004033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.005002 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:00.005015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.005874 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.005878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.005960 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.006420 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:00.006437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:00.007837 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:00.009915 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.009918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.009920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.010189 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.010498 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.010500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.010502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.010591 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.011024 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:00.011031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:00.011154 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 12:31:00.011241 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:31:00.011364 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:31:00.011930 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: Bootstrap starting.
I20260926 12:31:00.012063 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.001777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.7.99.60:42319)
0926 12:31:00.002066 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:00.002073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:00.002083 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:31:00.002109 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:00.002111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:00.002140 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:00.002247 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:00.002654 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.003920 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.005392 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.005745 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.006171 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.007929 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.007932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.007934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.007955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.009795 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.010314 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.010317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.010319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.010398 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.010701 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.010703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.010705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.010905 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.011687 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.011757 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.011861 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
Sep 26 12:31:00 dist-test-slave-3jw2 krb5kdc[20891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425859, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:31:00.014518 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: Bootstrap starting.
I20260926 12:31:00.015740 21123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:00.016319 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.005514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:42319 (local address 127.0.0.1:32996)
0926 12:31:00.005734 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:00.005747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:00.005825 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:00.007323 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:00.007327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:00.007502 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:00.007628 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:00.007635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.009217 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.009221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.010058 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:00.010067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.010675 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.010679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:00.010762 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.011197 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:00.011211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:00.012710 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:00.014722 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.014725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.014727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.014959 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.015263 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:00.015266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:00.015268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:00.015356 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:00.015782 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:00.015785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:00.015876 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:31:00.015984 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:31:00.016117 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 12:31:00.016362 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.005950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.7.99.60:42319)
0926 12:31:00.006809 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:00.006820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:00.006835 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:31:00.006872 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:00.006876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:00.006928 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:00.008210 (+ 1282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:00.008318 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.009081 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.010202 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.010547 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.010958 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.012852 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.012855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.012857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.012880 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.014592 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.015082 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.015086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.015088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.015167 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.015453 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.015456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.015458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.015667 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.015773 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.016074 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.016215 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":110,"threads_started":1}
I20260926 12:31:00.016834 21123 log.cc:824] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:00.017042 20972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:00.017864 20972 log.cc:824] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:00.018888 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: No bootstrap required, opened a new log
I20260926 12:31:00.019414 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: Bootstrap starting.
I20260926 12:31:00.020463 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: No bootstrap required, opened a new log
I20260926 12:31:00.021759 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:00.022579 21047 log.cc:824] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:00.023319 21123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.023588 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:31:00.023610 21123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:00.023663 21123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cf51d728b44e93954e4ab7ebea7de5, State: Initialized, Role: FOLLOWER
I20260926 12:31:00.024085 21123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.024544 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.024658 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.024726 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: No bootstrap required, opened a new log
I20260926 12:31:00.025405 21123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:00.027401 20972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.027647 20972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:00.027762 20972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b47a7c97f8e4e1e85db0dd73eded5b8, State: Initialized, Role: FOLLOWER
I20260926 12:31:00.028334 20972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.029726 20972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:31:00.030514 20925 tablet.cc:2369] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:31:00.031750 21151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:31:00.031884 21151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:31:00.029273 21054 master_runner.cc:186] Error getting master registration for 127.7.99.62:37373: OK, instance_id { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" instance_seqno: 1790425859533978 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:31:00.033175 21054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:31:00.034365 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.034664 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:00.034750 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e64a3faf9247f2bcd793094dc3c6a7, State: Initialized, Role: FOLLOWER
I20260926 12:31:00.035279 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.036685 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:00.037900 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.026524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.7.99.62:37373)
0926 12:31:00.026649 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:00.026653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:00.028121 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 12:31:00.028344 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:00.028347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:00.028395 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:00.028446 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:00.029465 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.030244 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.031383 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.031694 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.031777 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.034014 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.034017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.034020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.034044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.035754 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.036506 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.036509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.036511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.036605 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.036929 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.036934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.036938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.037178 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.037302 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.037639 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.037753 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:31:00.034737 20904 master_runner.cc:186] Error getting master registration for 127.7.99.61:37467: OK, instance_id { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" instance_seqno: 1790425859753525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:31:00.038968 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.039094 20904 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:31:00.039160 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 12:31:00.042239 21177 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 12:31:00.042256 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
W20260926 12:31:00.042369 21177 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:31:00.043184 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.044797 20979 master_runner.cc:263] No leader master found from master 36e64a3faf9247f2bcd793094dc3c6a7
I20260926 12:31:00.044854 20979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:31:00.045178 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.026071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.7.99.61:37467)
0926 12:31:00.026482 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:00.026487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:00.032002 (+ 5515us) server_negotiation.cc:407] Connection header received
0926 12:31:00.032190 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:00.032193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:00.032253 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:00.032304 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:00.033223 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.034021 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.035090 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.035398 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.035471 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.041211 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.041215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.041218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.041243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.043072 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.043739 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.043742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.043744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.043828 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.044144 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.044150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.044151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.044424 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.044557 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.044874 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.044991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
I20260926 12:31:00.047945 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:00.026226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.7.99.60:42319)
0926 12:31:00.026339 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:00.026343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:00.039613 (+ 13270us) server_negotiation.cc:407] Connection header received
0926 12:31:00.039775 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:00.039778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:00.039826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:00.039876 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:00.040773 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.041750 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.042980 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:00.043283 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:00.043396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:00.044782 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:00.044788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:00.044793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:00.044821 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:00.046372 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.046775 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.046778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.046780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.046858 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:00.047134 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:00.047137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:00.047138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:00.047303 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:00.047469 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:00.047690 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 12:31:00.047802 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 12:31:00.048400 21164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:31:00.048480 21164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 12:31:00.209748 21076 tablet.cc:2369] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:31:00.250149 21000 tablet.cc:2369] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:00.284194 21133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:00.284390 21133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.285305 21133 leader_election.cc:288] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b47a7c97f8e4e1e85db0dd73eded5b8 (127.7.99.62:37373), 36e64a3faf9247f2bcd793094dc3c6a7 (127.7.99.61:37467)
I20260926 12:31:00.285877 21020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93cf51d728b44e93954e4ab7ebea7de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" is_pre_election: true
I20260926 12:31:00.285876 20946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93cf51d728b44e93954e4ab7ebea7de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" is_pre_election: true
I20260926 12:31:00.286206 20946 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93cf51d728b44e93954e4ab7ebea7de5 in term 0.
I20260926 12:31:00.286206 21020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93cf51d728b44e93954e4ab7ebea7de5 in term 0.
I20260926 12:31:00.286662 21071 leader_election.cc:302] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b47a7c97f8e4e1e85db0dd73eded5b8, 93cf51d728b44e93954e4ab7ebea7de5; no voters:
I20260926 12:31:00.286913 21133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:31:00.287000 21133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:31:00.287041 21133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:00.288074 21133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.288443 21133 leader_election.cc:288] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [CANDIDATE]: Term 1 election: Requested vote from peers 6b47a7c97f8e4e1e85db0dd73eded5b8 (127.7.99.62:37373), 36e64a3faf9247f2bcd793094dc3c6a7 (127.7.99.61:37467)
I20260926 12:31:00.288733 20946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93cf51d728b44e93954e4ab7ebea7de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8"
I20260926 12:31:00.288836 21020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93cf51d728b44e93954e4ab7ebea7de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e64a3faf9247f2bcd793094dc3c6a7"
I20260926 12:31:00.288856 20946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:00.289000 21020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:00.289927 20946 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93cf51d728b44e93954e4ab7ebea7de5 in term 1.
I20260926 12:31:00.290056 21020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93cf51d728b44e93954e4ab7ebea7de5 in term 1.
I20260926 12:31:00.290272 21071 leader_election.cc:302] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b47a7c97f8e4e1e85db0dd73eded5b8, 93cf51d728b44e93954e4ab7ebea7de5; no voters:
I20260926 12:31:00.290457 21133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:00.290730 21133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [term 1 LEADER]: Becoming Leader. State: Replica: 93cf51d728b44e93954e4ab7ebea7de5, State: Running, Role: LEADER
I20260926 12:31:00.291085 21133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } }
I20260926 12:31:00.293079 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cf51d728b44e93954e4ab7ebea7de5. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.293195 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:00.293540 21181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:00.294365 21181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:00.298785 21020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [term 1 FOLLOWER]: Refusing update from remote peer 93cf51d728b44e93954e4ab7ebea7de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:00.298795 20946 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [term 1 FOLLOWER]: Refusing update from remote peer 93cf51d728b44e93954e4ab7ebea7de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:00.299492 21179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:00.299638 21133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:00.303773 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cf51d728b44e93954e4ab7ebea7de5. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.303781 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cf51d728b44e93954e4ab7ebea7de5. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.303892 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.303892 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.305956 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.306093 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:00.306249 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.306373 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:00.306834 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.306943 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.307030 21184 mvcc.cc:205] Tried to move back new op lower bound from 7333584323777298432 to 7333584323754971136. Current Snapshot: MvccSnapshot[applied={T|T < 7333584323777298432}]
I20260926 12:31:00.307077 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93cf51d728b44e93954e4ab7ebea7de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47a7c97f8e4e1e85db0dd73eded5b8" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 37373 } } peers { permanent_uuid: "36e64a3faf9247f2bcd793094dc3c6a7" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 37467 } } peers { permanent_uuid: "93cf51d728b44e93954e4ab7ebea7de5" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 42319 } } }
I20260926 12:31:00.307179 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:31:00.307490 21183 mvcc.cc:205] Tried to move back new op lower bound from 7333584323777298432 to 7333584323754971136. Current Snapshot: MvccSnapshot[applied={T|T < 7333584323777298432}]
I20260926 12:31:00.307706 21181 catalog_manager.cc:1347] Generated new cluster ID: 28d3ab88c2a64366ba6c9fc15088b3bf
I20260926 12:31:00.307766 21181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:00.314298 21181 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:31:00.315423 21181 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:00.321183 21181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93cf51d728b44e93954e4ab7ebea7de5: Generated new TSK 0
I20260926 12:31:00.321779 21181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:31:01.043618 21177 catalog_manager.cc:1259] Loaded cluster ID: 28d3ab88c2a64366ba6c9fc15088b3bf
I20260926 12:31:01.043732 21177 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: loading cluster ID for follower catalog manager: success
I20260926 12:31:01.045513 21177 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: acquiring CA information for follower catalog manager: success
I20260926 12:31:01.046687 21177 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36e64a3faf9247f2bcd793094dc3c6a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:31:01.049494 21164 catalog_manager.cc:1259] Loaded cluster ID: 28d3ab88c2a64366ba6c9fc15088b3bf
I20260926 12:31:01.049583 21164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: loading cluster ID for follower catalog manager: success
I20260926 12:31:01.051123 21164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: acquiring CA information for follower catalog manager: success
I20260926 12:31:01.052206 21164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b47a7c97f8e4e1e85db0dd73eded5b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:31:01.375249 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20904
I20260926 12:31:01.381386 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 20979
I20260926 12:31:01.387058 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21054
2026-09-26T12:31:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3176 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:31:01 dist-test-slave-3jw2 krb5kdc[21197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:31:01 dist-test-slave-3jw2 krb5kdc[21197](info): set up 2 sockets
Sep 26 12:31:01 dist-test-slave-3jw2 krb5kdc[21197](info): commencing operation
krb5kdc: starting...
W20260926 12:31:02.440672 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:31:02.560439 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40969
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:02.670269 21210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:02.670526 21210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:02.670595 21210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:02.674608 21210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:02.674664 21210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:02.674681 21210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:02.674697 21210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:02.674712 21210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:02.679953 21210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:02.681054 21210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:02.682094 21210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:02.690625 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:02.690680 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:02.690668 21210 server_base.cc:1034] running on GCE node
W20260926 12:31:02.690632 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:02.691510 21210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:02.692672 21210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:02.693903 21210 hybrid_clock.cc:648] HybridClock initialized: now 1790425862693867 us; error 55 us; skew 500 ppm
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425862, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:02.696691 21210 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:02.697582 21210 webserver.cc:460] Webserver started at http://127.7.99.62:35209/ using document root <none> and password file <none>
I20260926 12:31:02.698128 21210 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:02.698215 21210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:02.698469 21210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:02.700524 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f46c8e6629844467b9aab889e6562538"
format_stamp: "Formatted at 2026-09-26 12:31:02 on dist-test-slave-3jw2"
server_key: "ce02f1d30e6243a158bbb34e8ad05bc5"
server_key_iv: "3d3790806d328105336e2a54a93e3f2c"
server_key_version: "encryptionkey@0"
I20260926 12:31:02.701108 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f46c8e6629844467b9aab889e6562538"
format_stamp: "Formatted at 2026-09-26 12:31:02 on dist-test-slave-3jw2"
server_key: "ce02f1d30e6243a158bbb34e8ad05bc5"
server_key_iv: "3d3790806d328105336e2a54a93e3f2c"
server_key_version: "encryptionkey@0"
I20260926 12:31:02.704640 21210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:31:02.706905 21225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:02.708365 21210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:31:02.708495 21210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f46c8e6629844467b9aab889e6562538"
format_stamp: "Formatted at 2026-09-26 12:31:02 on dist-test-slave-3jw2"
server_key: "ce02f1d30e6243a158bbb34e8ad05bc5"
server_key_iv: "3d3790806d328105336e2a54a93e3f2c"
server_key_version: "encryptionkey@0"
I20260926 12:31:02.708676 21210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:02.726603 21210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:02.729357 21210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:02.729555 21210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:02.736033 21210 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:40969
I20260926 12:31:02.736094 21277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:40969 every 8 connection(s)
I20260926 12:31:02.737052 21210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:02.740154 21278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:02.745585 21278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538: Bootstrap starting.
I20260926 12:31:02.746031 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21210
I20260926 12:31:02.746124 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:31:02.746411 7564 external_mini_cluster.cc:1428] Setting key e428dbf92448698b72919964a0fa71ef
I20260926 12:31:02.748232 21278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:02.749120 21278 log.cc:824] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:02.750931 21278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538: No bootstrap required, opened a new log
I20260926 12:31:02.754347 21278 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } }
I20260926 12:31:02.754585 21278 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:02.754662 21278 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46c8e6629844467b9aab889e6562538, State: Initialized, Role: FOLLOWER
I20260926 12:31:02.755155 21278 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } }
I20260926 12:31:02.755316 21278 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:02.755388 21278 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:02.755487 21278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:02.756505 21278 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } }
I20260926 12:31:02.756867 21278 leader_election.cc:302] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46c8e6629844467b9aab889e6562538; no voters:
I20260926 12:31:02.757252 21278 leader_election.cc:288] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:31:02.757350 21283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:02.757596 21283 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [term 1 LEADER]: Becoming Leader. State: Replica: f46c8e6629844467b9aab889e6562538, State: Running, Role: LEADER
I20260926 12:31:02.757922 21283 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } }
I20260926 12:31:02.758540 21278 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:02.759189 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46c8e6629844467b9aab889e6562538. Latest consensus state: current_term: 1 leader_uuid: "f46c8e6629844467b9aab889e6562538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } } }
I20260926 12:31:02.759521 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:02.759403 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46c8e6629844467b9aab889e6562538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c8e6629844467b9aab889e6562538" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 40969 } } }
I20260926 12:31:02.759809 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:02.760109 21292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:02.762159 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:02.747692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.7.99.62:40969)
0926 12:31:02.748111 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:02.748131 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:02.748201 (+ 70us) server_negotiation.cc:407] Connection header received
0926 12:31:02.749077 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:02.749100 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:02.749525 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:02.749895 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:02.750714 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:02.751738 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:02.752670 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:02.753158 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:02.753348 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:02.756151 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:02.756202 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:02.756222 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:02.756254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:02.758343 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:02.758844 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:02.758851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:02.758856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:02.759034 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:02.759365 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:02.759368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:02.759370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:02.759960 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:02.760185 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:02.760929 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 12:31:02.761223 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260926 12:31:02.763269 21292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:02.765412 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:31:02.769483 21292 catalog_manager.cc:1347] Generated new cluster ID: 2f0bc03cb3604d5eade60c012a9f8f04
I20260926 12:31:02.769550 21292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:02.787338 21292 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:31:02.788653 21292 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:02.796674 21292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46c8e6629844467b9aab889e6562538: Generated new TSK 0
I20260926 12:31:02.797472 21292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:31:02.821117 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21210
2026-09-26T12:31:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1432 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21305](info): set up 2 sockets
Sep 26 12:31:02 dist-test-slave-3jw2 krb5kdc[21305](info): commencing operation
krb5kdc: starting...
W20260926 12:31:03.856730 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:31:03 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:31:03.975287 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:04.083825 21318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:04.084127 21318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:04.084230 21318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:04.088243 21318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:04.088320 21318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:04.088359 21318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:04.088428 21318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:04.088470 21318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:04.093695 21318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:36533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:04.094691 21318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:04.095687 21318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:04.104249 21323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.104221 21326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.104470 21324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.104429 21318 server_base.cc:1034] running on GCE node
I20260926 12:31:04.104996 21318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:04.106042 21318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:04.107206 21318 hybrid_clock.cc:648] HybridClock initialized: now 1790425864107189 us; error 29 us; skew 500 ppm
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:04.115844 21318 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:04.116870 21318 webserver.cc:460] Webserver started at http://127.7.99.62:42009/ using document root <none> and password file <none>
I20260926 12:31:04.117435 21318 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:04.117528 21318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:04.117769 21318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:04.119798 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "2a9c84b90894466e8bb47c14c092dc92"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3064e06bd927e71cb42734715920044a"
server_key_iv: "7da5572a7ebadafed92cd22fa9807ccc"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.120391 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "2a9c84b90894466e8bb47c14c092dc92"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3064e06bd927e71cb42734715920044a"
server_key_iv: "7da5572a7ebadafed92cd22fa9807ccc"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.123976 21318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:31:04.126398 21333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.127548 21318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:04.127691 21318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "2a9c84b90894466e8bb47c14c092dc92"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3064e06bd927e71cb42734715920044a"
server_key_iv: "7da5572a7ebadafed92cd22fa9807ccc"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.127800 21318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:04.137265 21318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:04.139981 21318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:04.140228 21318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:04.147272 21318 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:36533
I20260926 12:31:04.147541 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:36533 every 8 connection(s)
I20260926 12:31:04.148319 21318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:04.151000 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:04.151645 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21318
I20260926 12:31:04.151793 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:31:04.152096 7564 external_mini_cluster.cc:1428] Setting key 1a4eca41f30dcd369e0d1e5b730a2e60
I20260926 12:31:04.157132 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap starting.
I20260926 12:31:04.159855 21386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:04.160786 21386 log.cc:824] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:04.163223 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: No bootstrap required, opened a new log
I20260926 12:31:04.166821 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.153445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.7.99.62:36533)
0926 12:31:04.153841 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.153856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.153892 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:31:04.154520 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.154545 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.154875 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.155157 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:04.156037 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.157088 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.158188 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.158664 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.158779 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.161760 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:04.161766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:04.161783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:04.161810 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:04.163744 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.164150 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.164156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.164182 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.164317 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.164585 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.164588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.164590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.165091 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:04.165302 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.165532 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.165857 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260926 12:31:04.167825 21386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:04.168121 21386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:04.168236 21386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Initialized, Role: FOLLOWER
I20260926 12:31:04.168668 21386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:04.168787 21386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:04.168871 21386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:04.168967 21386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:04.169894 21386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:04.170257 21386 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9c84b90894466e8bb47c14c092dc92; no voters:
I20260926 12:31:04.170580 21386 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:31:04.170715 21391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:04.170955 21391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 LEADER]: Becoming Leader. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Running, Role: LEADER
I20260926 12:31:04.171319 21391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:04.171808 21386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:04.173090 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:04.173249 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:04.173090 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9c84b90894466e8bb47c14c092dc92. Latest consensus state: current_term: 1 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:04.173367 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
W20260926 12:31:04.176777 21406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:31:04.176880 21406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:31:04.176968 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:04.177655 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:04.183467 21400 catalog_manager.cc:1347] Generated new cluster ID: c8ffae94870346a589a016a850a1fc61
I20260926 12:31:04.183543 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:04.197082 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:31:04.198313 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:04.205828 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Generated new TSK 0
I20260926 12:31:04.206528 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:31:04.244390 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:04.363647 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:04.363902 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:04.363945 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:04.367890 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:04.367942 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:04.367998 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:31:04.373440 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:04.374572 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:04.375564 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:04.384883 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.385008 21412 server_base.cc:1034] running on GCE node
W20260926 12:31:04.384893 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.385087 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.385681 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:04.386365 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:04.387552 21412 hybrid_clock.cc:648] HybridClock initialized: now 1790425864387523 us; error 34 us; skew 500 ppm
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:04.390295 21412 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:31:04.391094 21412 webserver.cc:460] Webserver started at http://127.7.99.1:45083/ using document root <none> and password file <none>
I20260926 12:31:04.391605 21412 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:04.391661 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:04.391876 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:04.393910 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "1246b953679c32a5c599862c2846aef8"
server_key_iv: "26bb9f1826879801ba74411116dedecf"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.394472 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "1246b953679c32a5c599862c2846aef8"
server_key_iv: "26bb9f1826879801ba74411116dedecf"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.398141 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:31:04.400525 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.401650 21412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:31:04.401767 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "1246b953679c32a5c599862c2846aef8"
server_key_iv: "26bb9f1826879801ba74411116dedecf"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.401871 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:04.415191 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:04.418089 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:04.418293 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:04.418898 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:04.419718 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:04.419768 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.419848 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:04.419878 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.428315 21412 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:34385
I20260926 12:31:04.428341 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:34385 every 8 connection(s)
I20260926 12:31:04.429188 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:31:04.430855 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21412
I20260926 12:31:04.431007 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:31:04.431360 7564 external_mini_cluster.cc:1428] Setting key 386c93794db6188fefb3ac06026c84d2
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:31:04.443580 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.430942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:36465 (local address 127.7.99.62:36533)
0926 12:31:04.431113 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.431119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.432321 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 12:31:04.433164 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.433168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.433223 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.433318 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:04.434435 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.435079 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.436946 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.437182 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.437270 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.440044 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:04.440050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:04.440053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:04.440085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:04.441728 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.442176 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.442179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.442180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.442249 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.442650 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.442653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.442654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.442823 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:04.442900 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.443347 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.443449 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:31:04.444681 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.431367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:36465)
0926 12:31:04.432125 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.432205 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.432965 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.433466 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.433472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.433813 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:04.434292 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.434305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.435256 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.435263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.436806 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.436821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.437344 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.437349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.437556 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.438217 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:04.438232 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:04.439850 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:04.441826 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.441831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.441842 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.442092 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.442326 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.442328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.442329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.442540 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.442914 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:04.442918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:04.443148 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.443300 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.443715 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":144,"threads_started":1}
I20260926 12:31:04.445885 21541 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:04.446123 21541 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:04.446609 21541 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:04.448045 21350 ts_manager.cc:194] Registered new tserver with Master: ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385)
I20260926 12:31:04.449168 21350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:36465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:31:04.459654 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:04.570331 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:04.570585 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:04.570626 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:04.574867 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:04.574920 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:04.574977 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:31:04.580010 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:04.581158 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:04.582178 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:04.591571 21554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.591565 21551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.591706 21546 server_base.cc:1034] running on GCE node
W20260926 12:31:04.591571 21552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.592247 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:04.592885 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:04.594046 21546 hybrid_clock.cc:648] HybridClock initialized: now 1790425864594027 us; error 26 us; skew 500 ppm
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:04.596776 21546 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:31:04.597558 21546 webserver.cc:460] Webserver started at http://127.7.99.2:45917/ using document root <none> and password file <none>
I20260926 12:31:04.598067 21546 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:04.598122 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:04.598281 21546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:04.600339 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "1154b6734b1b4bac8a42c01efa9e2403"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3633c8282bbf1a7571890ba9cbeeafc8"
server_key_iv: "494aa7a7f3707938f7c45ee1e3596cc6"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.600857 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "1154b6734b1b4bac8a42c01efa9e2403"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3633c8282bbf1a7571890ba9cbeeafc8"
server_key_iv: "494aa7a7f3707938f7c45ee1e3596cc6"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.604480 21546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:31:04.606917 21561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.608091 21546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:31:04.608233 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "1154b6734b1b4bac8a42c01efa9e2403"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3633c8282bbf1a7571890ba9cbeeafc8"
server_key_iv: "494aa7a7f3707938f7c45ee1e3596cc6"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.608342 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:04.629925 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:04.632874 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:04.633092 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:04.633674 21546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:04.634469 21546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:04.634519 21546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.634605 21546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:04.634649 21546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.643627 21546 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:34559
I20260926 12:31:04.643652 21674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:34559 every 8 connection(s)
I20260926 12:31:04.644549 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:31:04.645961 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21546
I20260926 12:31:04.646085 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:31:04.646363 7564 external_mini_cluster.cc:1428] Setting key 1c19e2020195305f5ba32183e1c485e2
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:31:04.658567 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.647836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:49169 (local address 127.7.99.62:36533)
0926 12:31:04.647972 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.647977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.647993 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:31:04.648577 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.648583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.648643 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.648736 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:04.649834 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.650724 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.651831 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.652059 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.652172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.654821 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:04.654829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:04.654831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:04.654852 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:04.656428 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.656917 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.656922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.656924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.657044 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.657409 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.657412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.657414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.657597 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:04.657701 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.658267 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.658395 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:31:04.659452 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.646765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:49169)
0926 12:31:04.647508 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.647561 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.648378 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.648886 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.648892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.649243 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:04.649693 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.649706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.650850 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.650853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.651699 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.651708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.652227 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.652232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.652398 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.653030 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:04.653045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:04.654664 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:04.656552 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.656558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.656572 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.656815 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.657161 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.657165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.657167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.657306 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.657700 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:04.657707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:04.657951 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.658221 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.658600 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":153,"threads_started":1}
I20260926 12:31:04.660668 21675 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:04.660935 21675 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:04.661554 21675 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:04.662685 21350 ts_manager.cc:194] Registered new tserver with Master: 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
I20260926 12:31:04.663260 21350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:49169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:31:04.674975 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:04.787917 21680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:04.788214 21680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:04.788305 21680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:04.792153 21680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:04.792229 21680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:04.792285 21680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:31:04.797284 21680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:04.798182 21680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:04.799105 21680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:04.808604 21685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.808593 21688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:04.808588 21686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:04.808761 21680 server_base.cc:1034] running on GCE node
I20260926 12:31:04.809273 21680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:04.809903 21680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:04.811087 21680 hybrid_clock.cc:648] HybridClock initialized: now 1790425864811057 us; error 41 us; skew 500 ppm
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:04.813773 21680 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:31:04.814567 21680 webserver.cc:460] Webserver started at http://127.7.99.3:34393/ using document root <none> and password file <none>
I20260926 12:31:04.815114 21680 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:04.815200 21680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:04.815423 21680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:04.817432 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "a5a6f3da34f64690abe951afba69f656"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "6d5f98d6c340caa7ccb67e84834d4588"
server_key_iv: "f393825362c13296bad580545937a0e8"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.817974 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "a5a6f3da34f64690abe951afba69f656"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "6d5f98d6c340caa7ccb67e84834d4588"
server_key_iv: "f393825362c13296bad580545937a0e8"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.821586 21680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:31:04.823839 21695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.824992 21680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:31:04.825137 21680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "a5a6f3da34f64690abe951afba69f656"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "6d5f98d6c340caa7ccb67e84834d4588"
server_key_iv: "f393825362c13296bad580545937a0e8"
server_key_version: "encryptionkey@0"
I20260926 12:31:04.825242 21680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:04.840340 21680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:04.843123 21680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:04.843411 21680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:04.843997 21680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:04.844823 21680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:04.844895 21680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.844959 21680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:04.845017 21680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:04.853569 21680 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:41873
I20260926 12:31:04.853608 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:41873 every 8 connection(s)
I20260926 12:31:04.854471 21680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:31:04.861681 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21680
I20260926 12:31:04.861805 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:31:04.862118 7564 external_mini_cluster.cc:1428] Setting key 4775b2fce96ae08de69c54aea9676fa2
Sep 26 12:31:04 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425864, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:04.868278 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.856276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:37607 (local address 127.7.99.62:36533)
0926 12:31:04.856415 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.856419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.857401 (+ 982us) server_negotiation.cc:407] Connection header received
0926 12:31:04.858276 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.858279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.858329 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.858411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:04.859530 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.860061 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.861205 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.861536 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.861649 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.864665 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:04.864671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:04.864678 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:04.864701 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:04.866190 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.866706 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.866714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.866716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.866787 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.867134 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.867137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.867139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.867335 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:04.867444 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.867944 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.868075 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:31:04.869065 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.856639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:37607)
0926 12:31:04.857275 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.857316 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.858064 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.858566 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.858572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.858910 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:04.859390 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.859403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.860211 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.860214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.861091 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.861100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.861679 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.861681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.861908 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.862592 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:04.862616 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:04.864485 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:04.866310 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.866316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.866331 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.866597 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.866886 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:04.866889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:04.866891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:04.867033 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:04.867441 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:04.867448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:04.867707 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.867888 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.868241 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":115,"threads_started":1}
I20260926 12:31:04.870107 21809 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:04.870368 21809 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:04.870848 21809 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:04.871831 21350 ts_manager.cc:194] Registered new tserver with Master: a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873)
I20260926 12:31:04.872527 21350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:37607
I20260926 12:31:04.876147 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:31:04.884932 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.878078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.7.99.62:36533)
0926 12:31:04.878194 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.878197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.878378 (+ 181us) server_negotiation.cc:407] Connection header received
0926 12:31:04.878463 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.878465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.878501 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.878568 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:04.879208 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.879694 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.880531 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.880735 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.880812 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.882054 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:04.882057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:04.882059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:04.882080 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:04.883496 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.883837 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.883839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.883840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.883908 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:04.884118 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:04.884120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:04.884121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:04.884418 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:04.884523 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.884722 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.884815 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:31:04.887729 21350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:31:04.890718 21350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:31:04.904040 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.898761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:41873 (local address 127.0.0.1:48260)
0926 12:31:04.899211 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.899255 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.899394 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.899946 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.899951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.899971 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:04.900130 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.900138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.901411 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.901416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.902309 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.902318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.903480 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.903484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.903611 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.903656 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.903769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.903891 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":79,"threads_started":1}
I20260926 12:31:04.904054 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.898892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.7.99.3:41873)
0926 12:31:04.899211 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.899222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.899479 (+ 257us) server_negotiation.cc:407] Connection header received
0926 12:31:04.899573 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.899581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.899806 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.899964 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:04.900286 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.901280 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.902501 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.903361 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.903481 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.903631 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.903798 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.903891 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":127,"threads_started":1}
I20260926 12:31:04.905200 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.899474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.0.0.1:38034)
0926 12:31:04.899711 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.899724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.899805 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.901057 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.901060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.901079 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:04.901180 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.901186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.902546 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.902549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.903738 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.903748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.904805 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.904809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.904904 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.904927 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.905005 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.905073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260926 12:31:04.905301 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.899671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.7.99.1:34385)
0926 12:31:04.900611 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.900625 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.900648 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:31:04.900721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.900729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.900900 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.901059 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:04.901275 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.902390 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.903870 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.904687 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.904792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.904925 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.905048 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.905150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":89,"threads_started":1}
I20260926 12:31:04.905625 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.899818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:34559 (local address 127.0.0.1:44100)
0926 12:31:04.900273 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:04.900286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:04.900370 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:04.901102 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:04.901107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:04.901125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:04.901252 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.901260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.903000 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.903003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.904115 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:04.904123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.905209 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.905213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:04.905298 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.905378 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 12:31:04.905437 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:31:04.905503 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":11,"thread_start_us":69,"threads_started":1}
I20260926 12:31:04.906297 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:04.900376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.7.99.2:34559)
0926 12:31:04.900708 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:04.900719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:04.900737 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:31:04.900801 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:04.900805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:04.900944 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:04.901596 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:04.901750 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.902860 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.904265 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:04.905080 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:04.905871 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:04.906024 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:04.906077 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 12:31:04.906162 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260926 12:31:04.907406 21743 tablet_service.cc:1467] Processing CreateTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 (DEFAULT_TABLE table=test-table [id=dde9d47eec754ca39838852b7e6dde47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:04.907788 21475 tablet_service.cc:1467] Processing CreateTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 (DEFAULT_TABLE table=test-table [id=dde9d47eec754ca39838852b7e6dde47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:04.908586 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c320dc9e206340fea812fd71cf8cc9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:04.908766 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c320dc9e206340fea812fd71cf8cc9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:04.908835 21609 tablet_service.cc:1467] Processing CreateTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 (DEFAULT_TABLE table=test-table [id=dde9d47eec754ca39838852b7e6dde47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:04.909804 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c320dc9e206340fea812fd71cf8cc9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:04.915189 21826 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap starting.
I20260926 12:31:04.916070 21827 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Bootstrap starting.
I20260926 12:31:04.916684 21828 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Bootstrap starting.
I20260926 12:31:04.917562 21826 tablet_bootstrap.cc:654] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:04.918321 21827 tablet_bootstrap.cc:654] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:04.918817 21826 log.cc:824] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:04.919175 21828 tablet_bootstrap.cc:654] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:04.919368 21827 log.cc:824] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:04.919981 21828 log.cc:824] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:04.921003 21826 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: No bootstrap required, opened a new log
I20260926 12:31:04.921196 21826 ts_tablet_manager.cc:1397] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:31:04.921520 21827 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: No bootstrap required, opened a new log
I20260926 12:31:04.921710 21827 ts_tablet_manager.cc:1397] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 12:31:04.921870 21828 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: No bootstrap required, opened a new log
I20260926 12:31:04.922057 21828 ts_tablet_manager.cc:1397] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:31:04.924558 21827 raft_consensus.cc:348] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.924799 21827 raft_consensus.cc:374] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:04.924858 21827 raft_consensus.cc:729] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a6f3da34f64690abe951afba69f656, State: Initialized, Role: FOLLOWER
I20260926 12:31:04.925071 21826 raft_consensus.cc:348] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.925314 21826 raft_consensus.cc:374] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:04.925369 21826 raft_consensus.cc:729] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca40eafe70d5473a9ad41dc62d0939f4, State: Initialized, Role: FOLLOWER
I20260926 12:31:04.925325 21827 consensus_queue.cc:260] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.925781 21826 consensus_queue.cc:260] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.925712 21828 raft_consensus.cc:348] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.925925 21809 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
I20260926 12:31:04.925938 21828 raft_consensus.cc:374] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:04.926049 21828 raft_consensus.cc:729] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1154b6734b1b4bac8a42c01efa9e2403, State: Initialized, Role: FOLLOWER
I20260926 12:31:04.926334 21827 ts_tablet_manager.cc:1428] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:31:04.926431 21828 consensus_queue.cc:260] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:04.926736 21826 ts_tablet_manager.cc:1428] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:31:04.927033 21541 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
I20260926 12:31:04.927281 21828 ts_tablet_manager.cc:1428] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:31:04.927786 21675 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
W20260926 12:31:04.930300 21542 tablet.cc:2369] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:05.037765 21832 raft_consensus.cc:482] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:05.038002 21832 raft_consensus.cc:504] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:05.039188 21832 leader_election.cc:288] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
I20260926 12:31:05.043510 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.040019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:44023 (local address 127.7.99.2:34559)
0926 12:31:05.040154 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.040159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.040329 (+ 170us) server_negotiation.cc:407] Connection header received
0926 12:31:05.040406 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.040409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.040468 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.040552 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:05.040790 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.041299 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.042508 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.042922 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.043041 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.043098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.043318 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.043381 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:31:05.043514 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.039515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.7.99.3:55881)
0926 12:31:05.039651 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.039669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.039771 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.040208 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.040211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.040232 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:05.040388 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.040394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.041246 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.041249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.042356 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.042366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.043043 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.043047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.043190 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.043211 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.043302 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.043375 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":85}
I20260926 12:31:05.043524 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.039585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:55881 (local address 127.7.99.1:34385)
0926 12:31:05.039715 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.039718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.039735 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:31:05.039895 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.039899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.039971 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.040084 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:05.040536 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.041141 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.042531 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.042921 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.043027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.043093 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.043320 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.043392 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:31:05.043533 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.039810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:34559 (local address 127.7.99.3:44023)
0926 12:31:05.040223 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.040236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.040333 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.040553 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.040556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.040573 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:05.040681 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.040686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.041423 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.041425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.042359 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.042373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.043066 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.043069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.043189 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.043213 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.043303 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.043383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":27,"thread_start_us":175,"threads_started":1}
I20260926 12:31:05.044353 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c320dc9e206340fea812fd71cf8cc9e6" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1154b6734b1b4bac8a42c01efa9e2403" is_pre_election: true
I20260926 12:31:05.044364 21495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c320dc9e206340fea812fd71cf8cc9e6" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" is_pre_election: true
I20260926 12:31:05.044667 21629 raft_consensus.cc:2454] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 0.
I20260926 12:31:05.044667 21495 raft_consensus.cc:2454] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 0.
I20260926 12:31:05.045154 21698 leader_election.cc:302] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a6f3da34f64690abe951afba69f656, ca40eafe70d5473a9ad41dc62d0939f4; no voters:
I20260926 12:31:05.045431 21832 raft_consensus.cc:2781] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:31:05.045506 21832 raft_consensus.cc:482] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:31:05.045540 21832 raft_consensus.cc:3037] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:05.046568 21832 raft_consensus.cc:504] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:05.046947 21832 leader_election.cc:288] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 election: Requested vote from peers ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
I20260926 12:31:05.047331 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c320dc9e206340fea812fd71cf8cc9e6" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1154b6734b1b4bac8a42c01efa9e2403"
I20260926 12:31:05.047385 21495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c320dc9e206340fea812fd71cf8cc9e6" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
I20260926 12:31:05.047505 21629 raft_consensus.cc:3037] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:05.047516 21495 raft_consensus.cc:3037] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:05.048604 21629 raft_consensus.cc:2454] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 1.
I20260926 12:31:05.048621 21495 raft_consensus.cc:2454] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 1.
I20260926 12:31:05.048918 21698 leader_election.cc:302] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a6f3da34f64690abe951afba69f656, ca40eafe70d5473a9ad41dc62d0939f4; no voters:
I20260926 12:31:05.049109 21832 raft_consensus.cc:2781] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:05.049319 21832 raft_consensus.cc:686] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 1 LEADER]: Becoming Leader. State: Replica: a5a6f3da34f64690abe951afba69f656, State: Running, Role: LEADER
I20260926 12:31:05.049677 21832 consensus_queue.cc:237] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:05.052901 21349 catalog_manager.cc:5495] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5a6f3da34f64690abe951afba69f656 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "a5a6f3da34f64690abe951afba69f656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } health_report { overall_health: UNKNOWN } } }
I20260926 12:31:05.092036 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.088860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.7.99.3:41873)
0926 12:31:05.088991 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.088995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.089017 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:31:05.089090 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.089092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.089145 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.089232 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:05.089431 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.089904 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.090719 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.090920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.090993 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.091234 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:05.091648 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:31:05.091732 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.091784 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.091828 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:31:05.097609 21629 raft_consensus.cc:1261] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Refusing update from remote peer a5a6f3da34f64690abe951afba69f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:05.097609 21495 raft_consensus.cc:1261] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Refusing update from remote peer a5a6f3da34f64690abe951afba69f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:05.098351 21832 consensus_queue.cc:1035] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:05.098572 21836 consensus_queue.cc:1035] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:05.106069 21839 mvcc.cc:205] Tried to move back new op lower bound from 7333584343432622080 to 7333584343246639104. Current Snapshot: MvccSnapshot[applied={T|T < 7333584343432622080}]
I20260926 12:31:05.109246 21840 mvcc.cc:205] Tried to move back new op lower bound from 7333584343432622080 to 7333584343246639104. Current Snapshot: MvccSnapshot[applied={T|T < 7333584343432622080}]
I20260926 12:31:05.115572 21348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:05.115754 21348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:05.118180 21348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Sending DeleteTablet for 3 replicas of tablet c320dc9e206340fea812fd71cf8cc9e6
I20260926 12:31:05.118877 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:05 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48260
I20260926 12:31:05.119125 21609 tablet_service.cc:1514] Processing DeleteTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:05 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:44100
I20260926 12:31:05.119432 21849 tablet_replica.cc:330] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: stopping tablet replica
I20260926 12:31:05.119380 21475 tablet_service.cc:1514] Processing DeleteTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:05 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:38034
I20260926 12:31:05.119575 21850 tablet_replica.cc:330] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: stopping tablet replica
I20260926 12:31:05.119844 21849 raft_consensus.cc:2229] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:31:05.119869 21850 raft_consensus.cc:2229] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:05.119998 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21412
I20260926 12:31:05.120223 21850 raft_consensus.cc:2258] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:05.120602 21849 raft_consensus.cc:2258] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:05.121938 21849 ts_tablet_manager.cc:1905] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:05.123136 21850 ts_tablet_manager.cc:1905] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:05.125742 21849 ts_tablet_manager.cc:1918] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:05.125859 21849 log.cc:1196] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/c320dc9e206340fea812fd71cf8cc9e6
I20260926 12:31:05.125988 21850 ts_tablet_manager.cc:1918] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:05.126133 21850 log.cc:1196] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/c320dc9e206340fea812fd71cf8cc9e6
I20260926 12:31:05.126199 21849 ts_tablet_manager.cc:1939] T c320dc9e206340fea812fd71cf8cc9e6 P a5a6f3da34f64690abe951afba69f656: Deleting consensus metadata
I20260926 12:31:05.126510 21850 ts_tablet_manager.cc:1939] T c320dc9e206340fea812fd71cf8cc9e6 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting consensus metadata
I20260926 12:31:05.127599 21336 catalog_manager.cc:4841] TS a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): tablet c320dc9e206340fea812fd71cf8cc9e6 (table test-table [id=dde9d47eec754ca39838852b7e6dde47]) successfully deleted
I20260926 12:31:05.127846 21336 catalog_manager.cc:4841] TS 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): tablet c320dc9e206340fea812fd71cf8cc9e6 (table test-table [id=dde9d47eec754ca39838852b7e6dde47]) successfully deleted
W20260926 12:31:05.129406 21336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:34385 (error 108)
I20260926 12:31:05.130237 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21546
I20260926 12:31:05.130549 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.130264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.0.0.1:38040)
0926 12:31:05.130375 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.130446 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 12:31:05.131151 21336 catalog_manager.cc:4584] TS ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c320dc9e206340fea812fd71cf8cc9e6: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34385: connect: Connection refused (error 111)
I20260926 12:31:05.136413 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21680
I20260926 12:31:05.142184 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21318
I20260926 12:31:05.149000 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--webserver_interface=127.7.99.62
--webserver_port=42009
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:05.258188 21853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:05.258469 21853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:05.258512 21853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:05.262415 21853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:05.262487 21853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:05.262506 21853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:05.262522 21853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:05.262537 21853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:05.267702 21853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:36533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:05.268703 21853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:05.269670 21853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:05.282497 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.282485 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.282480 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:05.282761 21853 server_base.cc:1034] running on GCE node
I20260926 12:31:05.283639 21853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:05.284883 21853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:05.286095 21853 hybrid_clock.cc:648] HybridClock initialized: now 1790425865286073 us; error 30 us; skew 500 ppm
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:05.289247 21853 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:05.290231 21853 webserver.cc:460] Webserver started at http://127.7.99.62:42009/ using document root <none> and password file <none>
I20260926 12:31:05.290856 21853 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:05.290969 21853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:05.295392 21853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:31:05.297641 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.298794 21853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:31:05.298935 21853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "2a9c84b90894466e8bb47c14c092dc92"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3064e06bd927e71cb42734715920044a"
server_key_iv: "7da5572a7ebadafed92cd22fa9807ccc"
server_key_version: "encryptionkey@0"
I20260926 12:31:05.299528 21853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:05.313228 21853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:05.315887 21853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:05.316100 21853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:05.323134 21853 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:36533
I20260926 12:31:05.323419 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:36533 every 8 connection(s)
I20260926 12:31:05.324244 21853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:05.325363 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21853
I20260926 12:31:05.325867 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:34385
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=45083
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:05.329658 21922 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:31:05.331329 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap starting.
I20260926 12:31:05.348412 21922 log.cc:824] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:05.356781 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:05.357173 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap complete.
I20260926 12:31:05.361176 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:05.361572 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Initialized, Role: FOLLOWER
I20260926 12:31:05.362046 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:05.362193 21922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:05.362277 21922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:05.362406 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:31:05.364290 21922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:05.364614 21922 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9c84b90894466e8bb47c14c092dc92; no voters:
I20260926 12:31:05.364974 21922 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:31:05.365058 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:31:05.365255 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 LEADER]: Becoming Leader. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Running, Role: LEADER
I20260926 12:31:05.365546 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:05.366181 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:05.366452 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9c84b90894466e8bb47c14c092dc92. Latest consensus state: current_term: 2 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:05.366580 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:05.366925 21930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:05.366950 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:05.367167 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:05.371574 21930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:05.372205 21930 catalog_manager.cc:1259] Loaded cluster ID: c8ffae94870346a589a016a850a1fc61
I20260926 12:31:05.372262 21930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:05.373656 21930 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 12:31:05.379562 21930 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 12:31:05.384622 21930 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:05.385319 21930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Loaded TSK: 0
I20260926 12:31:05.386471 21930 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:31:05.438395 21924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:05.438686 21924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:05.438760 21924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:05.442814 21924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:05.442886 21924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:05.442975 21924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:31:05.448141 21924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:34385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=45083
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:05.449182 21924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:05.450237 21924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:05.460690 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.460741 21950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.460696 21949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:05.460851 21924 server_base.cc:1034] running on GCE node
I20260926 12:31:05.461400 21924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:05.462061 21924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:05.463248 21924 hybrid_clock.cc:648] HybridClock initialized: now 1790425865463225 us; error 39 us; skew 500 ppm
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:05.465921 21924 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:31:05.466969 21924 webserver.cc:460] Webserver started at http://127.7.99.1:45083/ using document root <none> and password file <none>
I20260926 12:31:05.467583 21924 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:05.467696 21924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:05.471589 21924 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:05.473829 21959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.475098 21924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:31:05.475250 21924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "1246b953679c32a5c599862c2846aef8"
server_key_iv: "26bb9f1826879801ba74411116dedecf"
server_key_version: "encryptionkey@0"
I20260926 12:31:05.475795 21924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:05.495239 21924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:05.498343 21924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:05.498541 21924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:05.499130 21924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:05.500320 21966 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:31:05.504361 21924 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:31:05.504419 21924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260926 12:31:05.504487 21924 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:31:05.506078 21924 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:31:05.506134 21924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:31:05.506374 21966 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap starting.
I20260926 12:31:05.516944 21924 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:34385
I20260926 12:31:05.516968 22073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:34385 every 8 connection(s)
I20260926 12:31:05.518244 21924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:31:05.522486 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 21924
I20260926 12:31:05.522992 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:34559
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=45917
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:05.527281 21966 log.cc:824] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:05.535549 21966 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:05.535924 21966 tablet_bootstrap.cc:492] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap complete.
I20260926 12:31:05.537328 21966 ts_tablet_manager.cc:1397] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20260926 12:31:05.538210 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.520314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:51823)
0926 12:31:05.520880 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.520920 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.521720 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.522999 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.523008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.523843 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:05.524641 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.524670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.526573 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.526584 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.527924 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.527936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.529093 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.529096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.529280 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.530010 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:05.530031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:05.532076 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:05.534601 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.534607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.534621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.535039 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.535472 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.535475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.535477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.535644 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.536338 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:05.536347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:05.536654 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.536845 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.537195 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":180,"threads_started":1}
I20260926 12:31:05.538236 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.520284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51823 (local address 127.7.99.62:36533)
0926 12:31:05.520634 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.520650 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.521052 (+ 402us) server_negotiation.cc:407] Connection header received
0926 12:31:05.522125 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.522136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.522546 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.522914 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:05.524902 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.526388 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.528075 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.528920 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.529148 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.532320 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:05.532328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:05.532344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:05.532375 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:05.534462 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.535199 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.535205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.535211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.535365 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.535759 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.535762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.535764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.536219 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:05.536349 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.536994 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.537293 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":125,"threads_started":1}
I20260926 12:31:05.540285 21966 raft_consensus.cc:348] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:05.540831 21966 raft_consensus.cc:729] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca40eafe70d5473a9ad41dc62d0939f4, State: Initialized, Role: FOLLOWER
I20260926 12:31:05.541327 21966 consensus_queue.cc:260] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } }
I20260926 12:31:05.541508 22074 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:05.541749 22074 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:05.542275 21966 ts_tablet_manager.cc:1428] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:31:05.542285 22074 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:05.544706 21886 ts_manager.cc:194] Registered new tserver with Master: ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385)
I20260926 12:31:05.547952 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:51823
I20260926 12:31:05.550457 22074 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.1@KRBTEST.COM
I20260926 12:31:05.557924 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.547407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.0.0.1:51664)
0926 12:31:05.548090 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.548131 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.548338 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.548838 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.548844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.549210 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:05.549307 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.549314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.550293 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.550296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.551132 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.551142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.551922 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.551948 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.552047 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.552645 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:05.552657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:05.554199 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:05.556272 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.556288 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.556291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.556511 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.556845 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.556848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.556850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.556934 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.557427 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:05.557432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:05.557525 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.557633 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.557795 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":112,"threads_started":1}
I20260926 12:31:05.558054 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.547493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.7.99.1:34385)
0926 12:31:05.547827 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.547842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.548241 (+ 399us) server_negotiation.cc:407] Connection header received
0926 12:31:05.548505 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.548512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.548701 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.548835 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:05.549449 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.550167 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.551256 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.551732 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.551837 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.554340 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:05.554349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:05.554361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:05.554387 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:05.556116 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.556621 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.556627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.556629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.556722 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.557028 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.557031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.557033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.557320 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:05.557445 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.557732 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.557898 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260926 12:31:05.559006 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet c320dc9e206340fea812fd71cf8cc9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:05 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:51664
I20260926 12:31:05.559845 22083 tablet_replica.cc:330] stopping tablet replica
I20260926 12:31:05.560141 22083 raft_consensus.cc:2229] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:05.560438 22083 raft_consensus.cc:2258] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:05.561352 22083 ts_tablet_manager.cc:1905] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:05.564788 22083 ts_tablet_manager.cc:1918] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:05.564865 22083 log.cc:1196] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/c320dc9e206340fea812fd71cf8cc9e6
I20260926 12:31:05.565155 22083 ts_tablet_manager.cc:1939] T c320dc9e206340fea812fd71cf8cc9e6 P ca40eafe70d5473a9ad41dc62d0939f4: Deleting consensus metadata
I20260926 12:31:05.566924 21872 catalog_manager.cc:4841] TS ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385): tablet c320dc9e206340fea812fd71cf8cc9e6 (table test-table [id=dde9d47eec754ca39838852b7e6dde47]) successfully deleted
W20260926 12:31:05.641253 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:05.641503 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:05.641546 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:05.645516 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:05.645567 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:05.645622 22078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:31:05.650844 22078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=45917
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:05.651769 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:05.652711 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:05.661857 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:05.661906 22078 server_base.cc:1034] running on GCE node
W20260926 12:31:05.661847 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.661854 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:05.662510 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:05.663158 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:05.664335 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790425865664305 us; error 38 us; skew 500 ppm
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:05.667057 22078 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:31:05.667948 22078 webserver.cc:460] Webserver started at http://127.7.99.2:45917/ using document root <none> and password file <none>
I20260926 12:31:05.668532 22078 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:05.668605 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:05.672297 22078 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 12:31:05.674320 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.675681 22078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:05.675797 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "1154b6734b1b4bac8a42c01efa9e2403"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3633c8282bbf1a7571890ba9cbeeafc8"
server_key_iv: "494aa7a7f3707938f7c45ee1e3596cc6"
server_key_version: "encryptionkey@0"
I20260926 12:31:05.676546 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:05.691397 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:05.694132 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:05.694324 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:05.694896 22078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:05.695681 22078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:05.695727 22078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.695792 22078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:05.695828 22078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.704396 22078 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:34559
I20260926 12:31:05.704481 22211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:34559 every 8 connection(s)
I20260926 12:31:05.705412 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:31:05.710690 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22078
I20260926 12:31:05.711134 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:41873
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=34393
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:05.719781 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.707142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:36241 (local address 127.7.99.62:36533)
0926 12:31:05.707307 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.707322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.708368 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 12:31:05.709193 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.709196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.709246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.709317 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:05.710428 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.711007 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.712846 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.713271 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.713390 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.715739 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:05.715745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:05.715751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:05.715788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:05.717622 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.718101 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.718107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.718111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.718209 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.718524 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.718530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.718532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.718791 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:05.718902 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.719362 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.719534 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:31:05.720422 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.707583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:36241)
0926 12:31:05.708251 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.708291 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.709018 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.709476 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.709482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.709824 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:05.710306 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.710320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.711143 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.711150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.712701 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.712713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.713418 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.713421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.713571 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.714166 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:05.714182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:05.715559 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:05.717738 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.717743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.717755 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.718000 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.718309 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.718311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.718312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.718434 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.718901 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:05.718905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:05.719117 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.719262 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.719537 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":101,"threads_started":1}
I20260926 12:31:05.721630 22212 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:05.721905 22212 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:05.722545 22212 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:05.723675 21886 ts_manager.cc:194] Registered new tserver with Master: 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
I20260926 12:31:05.724294 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:36241
W20260926 12:31:05.822423 22215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:05.822674 22215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:05.822714 22215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:05.826634 22215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:05.826707 22215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:05.826789 22215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:31:05.831969 22215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:41873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=34393
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:05.833014 22215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:05.833992 22215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:05.843443 22221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.843492 22220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:05.843451 22223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:05.843511 22215 server_base.cc:1034] running on GCE node
I20260926 12:31:05.844130 22215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:05.844863 22215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:05.846055 22215 hybrid_clock.cc:648] HybridClock initialized: now 1790425865846044 us; error 28 us; skew 500 ppm
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:05.848873 22215 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:31:05.849779 22215 webserver.cc:460] Webserver started at http://127.7.99.3:34393/ using document root <none> and password file <none>
I20260926 12:31:05.850358 22215 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:05.850463 22215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:05.854199 22215 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 12:31:05.856130 22230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.857322 22215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:05.857445 22215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "a5a6f3da34f64690abe951afba69f656"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "6d5f98d6c340caa7ccb67e84834d4588"
server_key_iv: "f393825362c13296bad580545937a0e8"
server_key_version: "encryptionkey@0"
I20260926 12:31:05.857995 22215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:05.886345 22215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:05.889864 22215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:05.890074 22215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:05.890668 22215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:05.891521 22215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:05.891588 22215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.891676 22215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:05.891731 22215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:05.900367 22215 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:41873
I20260926 12:31:05.900430 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:41873 every 8 connection(s)
I20260926 12:31:05.901382 22215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:31:05.908396 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22215
Sep 26 12:31:05 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425865, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:05.916139 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.903218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:39091 (local address 127.7.99.62:36533)
0926 12:31:05.903355 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.903359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.904637 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 12:31:05.905541 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.905544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.905601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.905677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:05.906819 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.907347 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.908654 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.908986 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.909090 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.912190 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:05.912197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:05.912203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:05.912236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:05.913951 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.914482 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.914485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.914486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.914574 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.914940 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.914944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.914945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.915135 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:05.915237 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.915853 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.915985 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:31:05.917285 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.903565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:39091)
0926 12:31:05.904500 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.904541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.905331 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.905853 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.905860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.906216 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:05.906678 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.906692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.907501 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.907504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.908520 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.908532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.909134 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.909139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.909361 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.910062 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:05.910085 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:05.911989 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:05.914069 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.914073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.914085 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.914380 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.914666 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:05.914669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:05.914671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:05.914844 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:05.915265 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:05.915271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:05.915600 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.915813 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.916186 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":566,"threads_started":1}
I20260926 12:31:05.917626 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.909082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.7.99.62:36533)
0926 12:31:05.909292 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.909296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.909496 (+ 200us) server_negotiation.cc:407] Connection header received
0926 12:31:05.909679 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.909683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.909732 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.909826 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:05.910770 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.911609 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.912641 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.912853 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.912923 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.914360 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:05.914364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:05.914366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:05.914388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:05.916042 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.916507 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.916510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.916512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.916601 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:05.916868 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:05.916871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:05.916872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:05.917056 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:05.917247 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.917350 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.917466 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260926 12:31:05.918526 22344 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36533
I20260926 12:31:05.918764 22344 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:05.919317 22344 heartbeater.cc:511] Master 127.7.99.62:36533 requested a full tablet report, sending...
I20260926 12:31:05.920270 21886 ts_manager.cc:194] Registered new tserver with Master: a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873)
I20260926 12:31:05.920804 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:39091
I20260926 12:31:05.931049 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:31:05.935410 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.931730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.7.99.62:36533)
0926 12:31:05.931873 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.931877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.932019 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:31:05.932147 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.932149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.932296 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.932378 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:05.932613 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.933089 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.934036 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.934243 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.934318 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.934629 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:05.934969 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:31:05.935055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.935193 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.935237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:31:05.935976 21886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:31:05.939083 21886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:31:05.946925 22008 tablet_service.cc:1467] Processing CreateTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 (DEFAULT_TABLE table=test-table [id=f58506cffed743e692a4d0da999a7e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:05.947623 22008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:05.950350 21966 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap starting.
I20260926 12:31:05.950654 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.945540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:41873 (local address 127.0.0.1:55760)
0926 12:31:05.945667 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.945682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.945762 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.946729 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.946733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.946751 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:05.946853 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.946860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.948280 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.948284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.949366 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.949374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.950202 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.950205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.950323 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.950372 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.950452 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.950529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":76}
I20260926 12:31:05.951284 21966 tablet_bootstrap.cc:654] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:05.951423 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.946118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.7.99.2:34559)
0926 12:31:05.946581 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.946593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.946621 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:31:05.946693 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.946697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.946875 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.947017 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:05.947429 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.948630 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.950064 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.950862 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.950966 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.951101 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.951199 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.951287 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":112,"threads_started":1}
I20260926 12:31:05.951419 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.946018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:34559 (local address 127.0.0.1:57684)
0926 12:31:05.946273 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:05.946286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:05.946369 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:05.947187 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:05.947191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:05.947207 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:05.947297 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.947303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.948771 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.948774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.949929 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:05.949940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.950969 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.950973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:05.951070 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.951091 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:31:05.951171 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:31:05.951236 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 12:31:05.951884 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:05.945702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.7.99.3:41873)
0926 12:31:05.946217 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:05.946230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:05.946254 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:31:05.946335 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:05.946340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:05.946568 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:05.946751 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:05.946952 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.948108 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.949486 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:05.950072 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:05.950190 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:05.950328 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:05.951611 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0926 12:31:05.951737 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":176,"threads_started":1}
I20260926 12:31:05.952965 21966 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: No bootstrap required, opened a new log
I20260926 12:31:05.953078 21966 ts_tablet_manager.cc:1397] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:31:05.953452 21966 raft_consensus.cc:348] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.953577 21966 raft_consensus.cc:374] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:05.953624 21966 raft_consensus.cc:729] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca40eafe70d5473a9ad41dc62d0939f4, State: Initialized, Role: FOLLOWER
I20260926 12:31:05.953739 21966 consensus_queue.cc:260] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.954084 21966 ts_tablet_manager.cc:1428] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:31:05.954771 22278 tablet_service.cc:1467] Processing CreateTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 (DEFAULT_TABLE table=test-table [id=f58506cffed743e692a4d0da999a7e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:05.954770 22146 tablet_service.cc:1467] Processing CreateTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 (DEFAULT_TABLE table=test-table [id=f58506cffed743e692a4d0da999a7e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:05.955840 22278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:05.955833 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:05.963728 22358 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Bootstrap starting.
I20260926 12:31:05.964576 22357 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Bootstrap starting.
I20260926 12:31:05.966885 22357 tablet_bootstrap.cc:654] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:05.967707 22357 log.cc:824] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:05.969714 22357 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: No bootstrap required, opened a new log
I20260926 12:31:05.969911 22357 ts_tablet_manager.cc:1397] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:31:05.970144 22358 tablet_bootstrap.cc:654] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:05.971107 22358 log.cc:824] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:05.972903 22358 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: No bootstrap required, opened a new log
I20260926 12:31:05.973101 22358 ts_tablet_manager.cc:1397] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 12:31:05.973062 22357 raft_consensus.cc:348] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.973348 22357 raft_consensus.cc:374] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:05.973423 22357 raft_consensus.cc:729] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1154b6734b1b4bac8a42c01efa9e2403, State: Initialized, Role: FOLLOWER
I20260926 12:31:05.973825 22357 consensus_queue.cc:260] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.974728 22212 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
I20260926 12:31:05.974808 22357 ts_tablet_manager.cc:1428] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:31:05.976864 22358 raft_consensus.cc:348] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.977185 22358 raft_consensus.cc:374] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:05.977252 22358 raft_consensus.cc:729] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a6f3da34f64690abe951afba69f656, State: Initialized, Role: FOLLOWER
I20260926 12:31:05.977718 22358 consensus_queue.cc:260] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:05.978349 22344 heartbeater.cc:503] Master 127.7.99.62:36533 was elected leader, sending a full tablet report...
I20260926 12:31:05.978627 22358 ts_tablet_manager.cc:1428] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 12:31:06.019498 22075 tablet.cc:2369] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:06.046123 22362 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:06.046383 22362 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:06.047621 22362 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559), ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385)
I20260926 12:31:06.051364 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.047999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:47413 (local address 127.7.99.2:34559)
0926 12:31:06.048131 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:06.048135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:06.048152 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:31:06.048268 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:06.048271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:06.048322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:06.048398 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:06.048799 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.049310 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.050398 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.050816 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.050936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:06.051012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:06.051175 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:31:06.051244 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:31:06.051364 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.047883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:34559 (local address 127.7.99.3:47413)
0926 12:31:06.048026 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.048044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:06.048142 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:06.048525 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:06.048528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:06.048562 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:06.048673 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:06.048679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.049423 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.049429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:06.050276 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:06.050285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.050935 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.050938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:06.051046 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:06.051073 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:31:06.051160 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:31:06.051234 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 12:31:06.051828 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.048446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:58469 (local address 127.7.99.1:34385)
0926 12:31:06.048581 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:06.048585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:06.048718 (+ 133us) server_negotiation.cc:407] Connection header received
0926 12:31:06.048867 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:06.048870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:06.048933 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:06.049032 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:06.049305 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.049810 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.050841 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.051321 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.051445 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:06.051619 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:06.051656 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 12:31:06.051723 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:31:06.051834 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.048217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.7.99.3:58469)
0926 12:31:06.048619 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.048640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:06.048754 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:06.049064 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:06.049066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:06.049082 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:06.049197 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:06.049202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.049924 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.049928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:06.050721 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:06.050729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.051460 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.051464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:06.051554 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:06.051574 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:31:06.051638 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:31:06.051703 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":132,"threads_started":1}
I20260926 12:31:06.052083 22166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd31cdcc579c4c30ae90a0f1ba7bcb72" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1154b6734b1b4bac8a42c01efa9e2403" is_pre_election: true
I20260926 12:31:06.052423 22166 raft_consensus.cc:2454] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 0.
I20260926 12:31:06.052624 22028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd31cdcc579c4c30ae90a0f1ba7bcb72" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" is_pre_election: true
I20260926 12:31:06.052915 22028 raft_consensus.cc:2454] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 0.
I20260926 12:31:06.052958 22233 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656; no voters:
I20260926 12:31:06.053256 22362 raft_consensus.cc:2781] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:31:06.053333 22362 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:31:06.053363 22362 raft_consensus.cc:3037] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:06.054385 22362 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:06.054725 22362 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 election: Requested vote from peers 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559), ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385)
I20260926 12:31:06.055096 22166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd31cdcc579c4c30ae90a0f1ba7bcb72" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1154b6734b1b4bac8a42c01efa9e2403"
I20260926 12:31:06.055127 22028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd31cdcc579c4c30ae90a0f1ba7bcb72" candidate_uuid: "a5a6f3da34f64690abe951afba69f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
I20260926 12:31:06.055197 22166 raft_consensus.cc:3037] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:06.055260 22028 raft_consensus.cc:3037] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:06.056268 22028 raft_consensus.cc:2454] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 1.
I20260926 12:31:06.056298 22166 raft_consensus.cc:2454] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a6f3da34f64690abe951afba69f656 in term 1.
I20260926 12:31:06.056543 22233 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a6f3da34f64690abe951afba69f656, ca40eafe70d5473a9ad41dc62d0939f4; no voters:
I20260926 12:31:06.056757 22362 raft_consensus.cc:2781] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:06.056980 22362 raft_consensus.cc:686] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 1 LEADER]: Becoming Leader. State: Replica: a5a6f3da34f64690abe951afba69f656, State: Running, Role: LEADER
I20260926 12:31:06.057291 22362 consensus_queue.cc:237] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:06.060521 21886 catalog_manager.cc:5495] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5a6f3da34f64690abe951afba69f656 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "a5a6f3da34f64690abe951afba69f656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } health_report { overall_health: UNKNOWN } } }
I20260926 12:31:06.079049 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.075862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.7.99.3:41873)
0926 12:31:06.075996 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:06.076000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:06.076012 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:31:06.076112 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:06.076114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:06.076214 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:06.076301 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:06.076485 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.077021 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.077828 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:06.078030 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:06.078104 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:06.078330 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:06.078716 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:31:06.078799 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:06.078857 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 12:31:06.078932 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:31:06.084746 22028 raft_consensus.cc:1261] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Refusing update from remote peer a5a6f3da34f64690abe951afba69f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:06.084749 22166 raft_consensus.cc:1261] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Refusing update from remote peer a5a6f3da34f64690abe951afba69f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:06.085431 22364 consensus_queue.cc:1035] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:06.085599 22362 consensus_queue.cc:1035] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:06.091616 22366 mvcc.cc:205] Tried to move back new op lower bound from 7333584347476082688 to 7333584347373330432. Current Snapshot: MvccSnapshot[applied={T|T < 7333584347476082688}]
I20260926 12:31:06.099641 21885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:06.099777 21885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:06.102003 21885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Sending DeleteTablet for 3 replicas of tablet fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:06.102653 22278 tablet_service.cc:1514] Processing DeleteTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:06 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:55760
I20260926 12:31:06.102804 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:06 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:51664
I20260926 12:31:06.102759 22146 tablet_service.cc:1514] Processing DeleteTablet for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:06 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:57684
I20260926 12:31:06.103101 22376 tablet_replica.cc:330] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: stopping tablet replica
I20260926 12:31:06.103196 22378 tablet_replica.cc:330] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: stopping tablet replica
I20260926 12:31:06.103269 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21924
I20260926 12:31:06.103389 22376 raft_consensus.cc:2229] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:31:06.103931 22378 raft_consensus.cc:2229] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:06.104249 22378 raft_consensus.cc:2258] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:06.104341 22376 raft_consensus.cc:2258] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:06.105966 22378 ts_tablet_manager.cc:1905] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:06.105986 22376 ts_tablet_manager.cc:1905] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:06.108655 22378 ts_tablet_manager.cc:1918] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:06.108748 22378 log.cc:1196] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:06.109099 22378 ts_tablet_manager.cc:1939] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P 1154b6734b1b4bac8a42c01efa9e2403: Deleting consensus metadata
I20260926 12:31:06.109259 22376 ts_tablet_manager.cc:1918] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:06.109365 22376 log.cc:1196] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:06.109705 22376 ts_tablet_manager.cc:1939] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P a5a6f3da34f64690abe951afba69f656: Deleting consensus metadata
I20260926 12:31:06.110265 21872 catalog_manager.cc:4841] TS 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 (table test-table [id=f58506cffed743e692a4d0da999a7e39]) successfully deleted
I20260926 12:31:06.110836 21872 catalog_manager.cc:4841] TS a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): tablet fd31cdcc579c4c30ae90a0f1ba7bcb72 (table test-table [id=f58506cffed743e692a4d0da999a7e39]) successfully deleted
W20260926 12:31:06.113875 21872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:34385 (error 108)
I20260926 12:31:06.114367 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22078
I20260926 12:31:06.115535 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.115199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34385 (local address 127.0.0.1:51674)
0926 12:31:06.115385 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.115451 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 12:31:06.115785 21872 catalog_manager.cc:4584] TS ca40eafe70d5473a9ad41dc62d0939f4 (127.7.99.1:34385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd31cdcc579c4c30ae90a0f1ba7bcb72: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34385: connect: Connection refused (error 111)
I20260926 12:31:06.120409 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22215
I20260926 12:31:06.125941 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 21853
I20260926 12:31:06.132150 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--webserver_interface=127.7.99.62
--webserver_port=42009
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:06.242674 22380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:06.242941 22380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:06.243023 22380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:06.247229 22380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:06.247285 22380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:06.247305 22380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:06.247357 22380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:06.247386 22380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:06.252676 22380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:36533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:06.253703 22380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:06.254743 22380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:06.263367 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.263432 22380 server_base.cc:1034] running on GCE node
W20260926 12:31:06.263360 22388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.263360 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.264143 22380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:06.265237 22380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:06.266422 22380 hybrid_clock.cc:648] HybridClock initialized: now 1790425866266406 us; error 26 us; skew 500 ppm
Sep 26 12:31:06 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:06.269125 22380 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:06.269984 22380 webserver.cc:460] Webserver started at http://127.7.99.62:42009/ using document root <none> and password file <none>
I20260926 12:31:06.270542 22380 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:06.270638 22380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:06.274348 22380 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 12:31:06.276467 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.277752 22380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 12:31:06.277890 22380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "2a9c84b90894466e8bb47c14c092dc92"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3064e06bd927e71cb42734715920044a"
server_key_iv: "7da5572a7ebadafed92cd22fa9807ccc"
server_key_version: "encryptionkey@0"
I20260926 12:31:06.278414 22380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:06.287904 22380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:06.290484 22380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:06.290670 22380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:06.297431 22380 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:36533
I20260926 12:31:06.297699 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:36533 every 8 connection(s)
I20260926 12:31:06.298467 22380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:06.302580 22448 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:31:06.304121 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap starting.
I20260926 12:31:06.308583 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22380
I20260926 12:31:06.309087 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:34385
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=45083
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:06.322206 22448 log.cc:824] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:06.333453 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:06.333827 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92: Bootstrap complete.
I20260926 12:31:06.337764 22448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:06.338131 22448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Initialized, Role: FOLLOWER
I20260926 12:31:06.338645 22448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:06.338760 22448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:06.338827 22448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:06.338944 22448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:31:06.340807 22448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:06.341125 22448 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9c84b90894466e8bb47c14c092dc92; no voters:
I20260926 12:31:06.341457 22448 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:31:06.341686 22452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:31:06.342013 22452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [term 3 LEADER]: Becoming Leader. State: Replica: 2a9c84b90894466e8bb47c14c092dc92, State: Running, Role: LEADER
I20260926 12:31:06.342374 22452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } }
I20260926 12:31:06.342658 22448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:06.343838 22454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9c84b90894466e8bb47c14c092dc92. Latest consensus state: current_term: 3 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:06.343964 22454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:06.344226 22453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2a9c84b90894466e8bb47c14c092dc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9c84b90894466e8bb47c14c092dc92" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36533 } } }
I20260926 12:31:06.344332 22453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9c84b90894466e8bb47c14c092dc92 [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:06.344435 22461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:06.349753 22461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:06.350569 22461 catalog_manager.cc:1259] Loaded cluster ID: c8ffae94870346a589a016a850a1fc61
I20260926 12:31:06.350636 22461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 12:31:06.351838 22461 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fefd49889ad google::LogMessage::Fail() at ??:0
@ 0x7fefd498b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fefd49884de google::LogMessage::Flush() at ??:0
@ 0x7fefd4989a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fefdf2c7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fefdf2c8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fefdf2c5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fefdf309640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fefd6bda3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fefd6bdac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fefd6bdc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fefd6bcb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fefd820f6db start_thread at ??:0
@ 0x7fefd348661f clone at ??:0
W20260926 12:31:06.449625 22450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:06.449929 22450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:06.450016 22450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:06.454282 22450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:06.454365 22450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:06.454480 22450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:31:06.459864 22450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:34385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=45083
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:06.461002 22450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:06.462054 22450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:06.471596 22489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.471616 22490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.471776 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.472060 22450 server_base.cc:1034] running on GCE node
I20260926 12:31:06.472529 22450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:06.473166 22450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:06.474344 22450 hybrid_clock.cc:648] HybridClock initialized: now 1790425866474330 us; error 28 us; skew 500 ppm
Sep 26 12:31:06 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:06.477793 22450 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:31:06.478770 22450 webserver.cc:460] Webserver started at http://127.7.99.1:45083/ using document root <none> and password file <none>
I20260926 12:31:06.479324 22450 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:06.479423 22450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:06.483319 22450 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:06.485682 22499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.487049 22450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:31:06.487200 22450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "ca40eafe70d5473a9ad41dc62d0939f4"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "1246b953679c32a5c599862c2846aef8"
server_key_iv: "26bb9f1826879801ba74411116dedecf"
server_key_version: "encryptionkey@0"
I20260926 12:31:06.487753 22450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:06.498785 22450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:06.501931 22450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:06.502127 22450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:06.502702 22450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:06.503752 22506 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:31:06.509459 22450 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:31:06.509539 22450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 12:31:06.509635 22450 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:31:06.511265 22450 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:31:06.511338 22450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:31:06.511627 22506 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap starting.
I20260926 12:31:06.521986 22450 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:34385
I20260926 12:31:06.522058 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:34385 every 8 connection(s)
I20260926 12:31:06.522927 22450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:31:06.525950 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22450
I20260926 12:31:06.526372 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:34559
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=45917
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:06.534708 22506 log.cc:824] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:06.544497 22506 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:06.545112 22506 tablet_bootstrap.cc:492] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Bootstrap complete.
I20260926 12:31:06.546517 22506 ts_tablet_manager.cc:1397] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s
I20260926 12:31:06.550559 22506 raft_consensus.cc:348] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:06.551151 22506 raft_consensus.cc:729] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca40eafe70d5473a9ad41dc62d0939f4, State: Initialized, Role: FOLLOWER
I20260926 12:31:06.551795 22506 consensus_queue.cc:260] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:06.552961 22506 ts_tablet_manager.cc:1428] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260926 12:31:06.593472 22616 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:31:06.524863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:51933)
0926 12:31:06.525347 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.525400 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:06.527648 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:06.592494 (+ 64846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
W20260926 12:31:06.594149 22614 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:36533 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:36533: Client connection negotiation failed: client connection to 127.7.99.62:36533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:31:06.594892 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.594607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:54727)
0926 12:31:06.594747 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.594824 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 12:31:06.595587 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.595363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:51813)
0926 12:31:06.595477 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.595527 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 12:31:06.595790 22614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:31:06.647639 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:06.647878 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:06.647933 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:06.651804 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:06.651856 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:06.651911 22617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:31:06.656975 22617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=45917
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:06.657909 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:06.658857 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:06.668272 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.668268 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.668267 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.668653 22617 server_base.cc:1034] running on GCE node
I20260926 12:31:06.669042 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:06.669682 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:06.670838 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790425866670811 us; error 38 us; skew 500 ppm
Sep 26 12:31:06 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:06.673625 22617 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:31:06.674437 22617 webserver.cc:460] Webserver started at http://127.7.99.2:45917/ using document root <none> and password file <none>
I20260926 12:31:06.674978 22617 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:06.675036 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:06.679579 22617 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:31:06.681965 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.683156 22617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:06.683297 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "1154b6734b1b4bac8a42c01efa9e2403"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "3633c8282bbf1a7571890ba9cbeeafc8"
server_key_iv: "494aa7a7f3707938f7c45ee1e3596cc6"
server_key_version: "encryptionkey@0"
I20260926 12:31:06.683820 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:06.697595 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:06.700906 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:06.701115 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:06.701682 22617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:06.702483 22617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:06.702555 22617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.702625 22617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:06.702683 22617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.712781 22617 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:34559
I20260926 12:31:06.713299 22747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:34559 every 8 connection(s)
I20260926 12:31:06.714170 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:31:06.718246 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.716711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:60751)
0926 12:31:06.717466 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.717577 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":98,"threads_started":1}
W20260926 12:31:06.718663 22748 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:36533 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:36533: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
I20260926 12:31:06.719219 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22617
I20260926 12:31:06.719264 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.719011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:56373)
0926 12:31:06.719137 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.719194 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:31:06.719695 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:41873
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:36533
--webserver_port=34393
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:06.719967 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.719721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:60695)
0926 12:31:06.719838 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.719896 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:31:06.720136 22748 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:31:06.827644 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:06.827940 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:06.828015 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:06.832036 22751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:06.832126 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:06.832239 22751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:31:06.837335 22751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:41873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=34393
--tserver_master_addrs=127.7.99.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:06.838388 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:06.839382 22751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:06.849002 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:06.849002 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.849200 22751 server_base.cc:1034] running on GCE node
W20260926 12:31:06.849107 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:06.849647 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:06.850278 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:06.851432 22751 hybrid_clock.cc:648] HybridClock initialized: now 1790425866851410 us; error 58 us; skew 500 ppm
Sep 26 12:31:06 dist-test-slave-3jw2 krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:06.854065 22751 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:31:06.854828 22751 webserver.cc:460] Webserver started at http://127.7.99.3:34393/ using document root <none> and password file <none>
I20260926 12:31:06.855336 22751 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:06.855394 22751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:06.859073 22751 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:06.861037 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.862226 22751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:06.862342 22751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "a5a6f3da34f64690abe951afba69f656"
format_stamp: "Formatted at 2026-09-26 12:31:04 on dist-test-slave-3jw2"
server_key: "6d5f98d6c340caa7ccb67e84834d4588"
server_key_iv: "f393825362c13296bad580545937a0e8"
server_key_version: "encryptionkey@0"
I20260926 12:31:06.862867 22751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:06.877882 22751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:06.881186 22751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:06.881397 22751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:06.882007 22751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:06.883236 22751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:06.883291 22751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.883461 22751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:06.883499 22751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:06.893460 22751 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:41873
I20260926 12:31:06.893484 22879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:41873 every 8 connection(s)
I20260926 12:31:06.894523 22751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:31:06.896102 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22751
I20260926 12:31:06.898550 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.896811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:60397)
0926 12:31:06.897752 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.897852 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":618,"threads_started":1}
W20260926 12:31:06.898953 22880 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:36533 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:36533: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
I20260926 12:31:06.899513 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.899320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:35583)
0926 12:31:06.899420 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.899463 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 12:31:06.900023 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:06.899856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:56849)
0926 12:31:06.899943 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:06.899981 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 12:31:06.900229 22880 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:31:07.596858 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:07.596457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:40169)
0926 12:31:07.596691 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:07.596747 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260926 12:31:07.720959 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:07.720616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:34573)
0926 12:31:07.720824 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:07.720870 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":62,"threads_started":1}
I20260926 12:31:07.901046 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:07.900689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:34049)
0926 12:31:07.900900 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:07.900948 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260926 12:31:08.301791 22887 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:08.302052 22887 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:08.303105 22887 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
Sep 26 12:31:08 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.3@KRBTEST.COM
Sep 26 12:31:08 dist-test-slave-3jw2 krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425866, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.2@KRBTEST.COM
I20260926 12:31:08.317492 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.303855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:44585 (local address 127.7.99.2:34559)
0926 12:31:08.304250 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:08.304261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:08.304291 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:31:08.304900 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:08.304905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:08.305216 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:08.305517 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:08.306755 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.307933 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309121 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.309558 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309729 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:08.312704 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:08.312713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:08.312730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:08.312772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:08.314620 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:08.315069 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:08.315077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:08.315081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:08.315195 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:08.315610 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:08.315614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:08.315616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:08.316131 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:08.316281 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:08.317029 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 12:31:08.317303 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260926 12:31:08.317487 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.303515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:52619 (local address 127.7.99.3:41873)
0926 12:31:08.303838 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:08.303866 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:08.304009 (+ 143us) server_negotiation.cc:407] Connection header received
0926 12:31:08.304883 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:08.304888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:08.305216 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:08.305519 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:08.306743 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.307935 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309112 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.309554 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309722 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:08.312381 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:08.312386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:08.312397 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:08.312423 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:08.314401 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:08.314992 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:08.314998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:08.315003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:08.315165 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:08.315610 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:08.315614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:08.315616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:08.316131 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:08.316281 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:08.317023 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 12:31:08.317303 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260926 12:31:08.317474 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.303338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:41873 (local address 127.7.99.1:52619)
0926 12:31:08.303882 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:08.303896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:08.304027 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:08.305563 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:08.305569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:08.306135 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:08.306584 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:08.306598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.308080 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.308083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:08.308985 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:08.308994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.309672 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:08.309896 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:08.310528 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:08.310543 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:08.312104 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:08.314516 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:08.314520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:08.314537 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:08.314883 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:08.315282 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:08.315285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:08.315287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:08.315493 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:08.316270 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:08.316278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:08.316664 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 12:31:08.316891 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 12:31:08.317301 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":337,"spinlock_wait_cycles":36352,"thread_start_us":85,"threads_started":1}
I20260926 12:31:08.317476 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.303686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:34559 (local address 127.7.99.1:44585)
0926 12:31:08.303968 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:08.303979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:08.304058 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:08.305562 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:08.305568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:08.306136 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:08.306583 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:08.306599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.308077 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.308080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:08.308988 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:08.308997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:08.309683 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:08.309686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:08.309895 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:08.310588 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:08.310600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:08.312510 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:08.314720 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:08.314725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:08.314727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:08.314966 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:08.315282 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:08.315285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:08.315287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:08.315493 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:08.316270 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:08.316278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:08.316664 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 12:31:08.316891 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 12:31:08.317251 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":189,"spinlock_wait_cycles":27648,"thread_start_us":67,"threads_started":1}
W20260926 12:31:08.320469 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
W20260926 12:31:08.320690 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:08.320756 22502 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca40eafe70d5473a9ad41dc62d0939f4; no voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656
I20260926 12:31:08.321059 22887 raft_consensus.cc:2726] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:31:08.597883 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.597605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:33217)
0926 12:31:08.597739 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:08.597797 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 12:31:08.722054 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.721639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:39203)
0926 12:31:08.721888 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:08.721938 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260926 12:31:08.902132 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:08.901743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:60463)
0926 12:31:08.901977 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:08.902037 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 12:31:09.599355 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:09.598817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:55235)
0926 12:31:09.599187 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:09.599253 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
I20260926 12:31:09.723289 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:09.722777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:46055)
0926 12:31:09.723111 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:09.723193 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":74,"threads_started":1}
I20260926 12:31:09.903337 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:09.902878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:52207)
0926 12:31:09.903159 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:09.903222 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260926 12:31:10.275413 22897 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:10.275578 22897 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:10.276042 22897 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
W20260926 12:31:10.276880 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
W20260926 12:31:10.277035 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:10.277091 22502 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca40eafe70d5473a9ad41dc62d0939f4; no voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656
I20260926 12:31:10.277230 22897 raft_consensus.cc:2726] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:31:10.600709 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:10.600290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:40427)
0926 12:31:10.600549 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:10.600612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260926 12:31:10.724457 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:10.724020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:35557)
0926 12:31:10.724308 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:10.724357 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260926 12:31:10.904448 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:10.904061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:47553)
0926 12:31:10.904316 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:10.904363 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260926 12:31:11.602146 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:11.601569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:58301)
0926 12:31:11.601974 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:11.602049 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":74,"threads_started":1}
I20260926 12:31:11.725554 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:11.725176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:58751)
0926 12:31:11.725407 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:11.725460 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260926 12:31:11.836794 22903 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:11.836936 22903 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:11.837571 22903 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
W20260926 12:31:11.838465 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
W20260926 12:31:11.838641 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:11.838718 22502 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca40eafe70d5473a9ad41dc62d0939f4; no voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656
I20260926 12:31:11.838905 22903 raft_consensus.cc:2726] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:31:11.905625 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:11.905201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:56505)
0926 12:31:11.905482 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:11.905536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260926 12:31:12.603318 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:12.602886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:52193)
0926 12:31:12.603152 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:12.603212 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260926 12:31:12.726809 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:12.726417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:40141)
0926 12:31:12.726669 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:12.726719 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260926 12:31:12.906885 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:12.906396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:49539)
0926 12:31:12.906710 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:12.906777 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260926 12:31:13.604635 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:13.604133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:36609)
0926 12:31:13.604481 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:13.604537 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260926 12:31:13.727917 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:13.727496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:54097)
0926 12:31:13.727775 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:13.727827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260926 12:31:13.908471 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:13.907808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:56145)
0926 12:31:13.908100 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:13.908309 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":71,"threads_started":1}
I20260926 12:31:14.115749 22911 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:14.115873 22911 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:14.116230 22911 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
W20260926 12:31:14.117076 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
W20260926 12:31:14.117287 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:14.117398 22502 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca40eafe70d5473a9ad41dc62d0939f4; no voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656
I20260926 12:31:14.117602 22911 raft_consensus.cc:2726] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:31:14.605795 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:14.605362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:34531)
0926 12:31:14.605626 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:14.605693 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260926 12:31:14.729099 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:14.728704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:46659)
0926 12:31:14.728940 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:14.728993 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260926 12:31:14.909737 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:14.909325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:45149)
0926 12:31:14.909571 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:14.909633 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260926 12:31:15.607045 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:15.606628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:54993)
0926 12:31:15.606889 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:15.606946 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260926 12:31:15.730305 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:15.729889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:47303)
0926 12:31:15.730156 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:15.730212 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260926 12:31:15.910918 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:15.910524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:57151)
0926 12:31:15.910757 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:15.910810 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260926 12:31:16.608390 22918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:16.607895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:38405)
0926 12:31:16.608202 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:16.608290 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
I20260926 12:31:16.731480 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:16.731078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:34445)
0926 12:31:16.731326 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:16.731377 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260926 12:31:16.912041 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:16.911661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:57639)
0926 12:31:16.911891 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:16.911946 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260926 12:31:17.609512 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:17.609119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:42155)
0926 12:31:17.609358 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:17.609414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260926 12:31:17.732836 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:17.732306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:41513)
0926 12:31:17.732652 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:17.732731 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":88,"threads_started":1}
I20260926 12:31:17.913249 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:17.912799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:43689)
0926 12:31:17.913099 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:17.913157 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260926 12:31:18.611157 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:18.610618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:46573)
0926 12:31:18.610938 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:18.610993 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":138,"threads_started":1}
I20260926 12:31:18.733955 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:18.733568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:38349)
0926 12:31:18.733800 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:18.733860 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260926 12:31:18.914570 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:18.914060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:47891)
0926 12:31:18.914345 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:18.914441 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260926 12:31:19.612382 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:19.611894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:51765)
0926 12:31:19.612156 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:19.612284 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260926 12:31:19.735081 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:19.734679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:37309)
0926 12:31:19.734930 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:19.734985 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260926 12:31:19.773911 22929 raft_consensus.cc:482] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:19.774035 22929 raft_consensus.cc:504] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a6f3da34f64690abe951afba69f656" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 41873 } } peers { permanent_uuid: "1154b6734b1b4bac8a42c01efa9e2403" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 34559 } } peers { permanent_uuid: "ca40eafe70d5473a9ad41dc62d0939f4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34385 } }
I20260926 12:31:19.774493 22929 leader_election.cc:288] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873), 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559)
W20260926 12:31:19.775269 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5a6f3da34f64690abe951afba69f656 (127.7.99.3:41873): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
W20260926 12:31:19.775417 22502 leader_election.cc:341] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1154b6734b1b4bac8a42c01efa9e2403 (127.7.99.2:34559): Not found: Tablet not found: fd31cdcc579c4c30ae90a0f1ba7bcb72
I20260926 12:31:19.775477 22502 leader_election.cc:302] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca40eafe70d5473a9ad41dc62d0939f4; no voters: 1154b6734b1b4bac8a42c01efa9e2403, a5a6f3da34f64690abe951afba69f656
I20260926 12:31:19.775640 22929 raft_consensus.cc:2726] T fd31cdcc579c4c30ae90a0f1ba7bcb72 P ca40eafe70d5473a9ad41dc62d0939f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:31:19.915904 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:19.915488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:53499)
0926 12:31:19.915716 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:19.915784 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260926 12:31:20.613667 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:20.613141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:60969)
0926 12:31:20.613466 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:20.613548 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260926 12:31:20.736109 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:20.735723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:36649)
0926 12:31:20.735962 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:20.736013 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260926 12:31:20.917167 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:20.916720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.3:40565)
0926 12:31:20.917014 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:20.917073 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260926 12:31:21.615176 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:21.614736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.1:38163)
0926 12:31:21.615027 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:21.615080 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260926 12:31:21.737391 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:21.736964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36533 (local address 127.7.99.2:35769)
0926 12:31:21.737224 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:21.737285 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 12:31:21.906872 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22450
I20260926 12:31:21.912405 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22617
I20260926 12:31:21.917527 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22751
2026-09-26T12:31:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19098 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:31:21 dist-test-slave-3jw2 krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:31:21 dist-test-slave-3jw2 krb5kdc[22938](info): set up 2 sockets
Sep 26 12:31:21 dist-test-slave-3jw2 krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260926 12:31:22.960975 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:31:23.079098 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:23.187578 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:23.187867 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:23.187939 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:23.192003 22951 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:31:23.192075 22951 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:31:23.192101 22951 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:31:23.192132 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:23.192184 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:23.192240 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:23.192276 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:23.192318 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:23.197471 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:23.198522 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:23.199558 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:23.208148 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.208154 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:23.208271 22951 server_base.cc:1034] running on GCE node
W20260926 12:31:23.208211 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:23.209021 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:23.210094 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:23.211282 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790425883211261 us; error 32 us; skew 500 ppm
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:23.220428 22951 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:23.221314 22951 webserver.cc:460] Webserver started at http://127.7.99.62:40647/ using document root <none> and password file <none>
I20260926 12:31:23.221872 22951 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:23.221971 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:23.222184 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:23.224143 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f4749c34cf624a089da3e339995b371f"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "995d35ffcc76ada1647d6d77bf8cfac0"
server_key_iv: "852444c6e5a13f12319fe1ad3ede1f00"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.224799 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f4749c34cf624a089da3e339995b371f"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "995d35ffcc76ada1647d6d77bf8cfac0"
server_key_iv: "852444c6e5a13f12319fe1ad3ede1f00"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.228451 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:31:23.230803 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.231981 22951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:23.232105 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f4749c34cf624a089da3e339995b371f"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "995d35ffcc76ada1647d6d77bf8cfac0"
server_key_iv: "852444c6e5a13f12319fe1ad3ede1f00"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.232241 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:23.267791 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:23.270723 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:23.270938 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:23.277907 22951 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44559
I20260926 12:31:23.278177 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44559 every 8 connection(s)
I20260926 12:31:23.278869 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:23.281670 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:23.285363 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 22951
I20260926 12:31:23.285534 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:31:23.285854 7564 external_mini_cluster.cc:1428] Setting key b3771fd5e65c878b4e57475d95a6d0ea
I20260926 12:31:23.287487 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap starting.
I20260926 12:31:23.289923 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:23.290658 23019 log.cc:824] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:23.292688 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: No bootstrap required, opened a new log
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:23.296876 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:23.297199 23019 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:23.297274 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4749c34cf624a089da3e339995b371f, State: Initialized, Role: FOLLOWER
I20260926 12:31:23.297789 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:23.297963 23019 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:23.298096 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:23.298204 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:23.299420 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:23.299810 23019 leader_election.cc:302] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4749c34cf624a089da3e339995b371f; no voters:
I20260926 12:31:23.300194 23019 leader_election.cc:288] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:31:23.300318 23024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:23.300557 23024 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 LEADER]: Becoming Leader. State: Replica: f4749c34cf624a089da3e339995b371f, State: Running, Role: LEADER
I20260926 12:31:23.300935 23024 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:23.301258 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:23.287299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.7.99.62:44559)
0926 12:31:23.287697 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:23.287711 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:23.287747 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:31:23.288466 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:23.288483 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:23.288795 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:23.289074 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:23.289961 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.291020 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.291942 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.292639 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.292802 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:23.295227 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:23.295236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:23.295257 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:23.295301 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:23.297901 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.298466 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.298472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.298478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.298614 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.298874 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.298877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.298878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.299434 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:23.299618 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:23.299901 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:31:23.300240 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":127,"threads_started":1}
I20260926 12:31:23.301462 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:23.302199 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4749c34cf624a089da3e339995b371f. Latest consensus state: current_term: 1 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:31:23.302407 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:23.302631 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:31:23.304049 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
W20260926 12:31:23.306214 23038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:31:23.306295 23038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:31:23.306370 23039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:23.307061 23039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:23.312274 23039 catalog_manager.cc:1347] Generated new cluster ID: 620d2bfe0fc24bb4ac1b88b996771247
I20260926 12:31:23.312340 23039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:23.336712 23039 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:31:23.338007 23039 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:23.348234 23039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Generated new TSK 0
I20260926 12:31:23.349103 23039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:31:23.380386 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:23.491534 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:23.491813 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:23.491880 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:23.495936 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:23.496008 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:23.496088 23045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:31:23.501283 23045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:23.502333 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:23.503386 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:23.512737 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.512811 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.512780 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:23.512976 23045 server_base.cc:1034] running on GCE node
I20260926 12:31:23.513417 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:23.514065 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:23.515218 23045 hybrid_clock.cc:648] HybridClock initialized: now 1790425883515189 us; error 55 us; skew 500 ppm
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:23.517896 23045 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:31:23.518707 23045 webserver.cc:460] Webserver started at http://127.7.99.1:37679/ using document root <none> and password file <none>
I20260926 12:31:23.519253 23045 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:23.519337 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:23.519557 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:23.521579 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "a3d79df178a64f85b88024b1c3d71f66"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "75e37c6417de9cdfe66a8a4ae7604781"
server_key_iv: "67837abef05d64a739b5795ab624a514"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.522161 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "a3d79df178a64f85b88024b1c3d71f66"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "75e37c6417de9cdfe66a8a4ae7604781"
server_key_iv: "67837abef05d64a739b5795ab624a514"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.525750 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:31:23.528239 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.529497 23045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:23.529619 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "a3d79df178a64f85b88024b1c3d71f66"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "75e37c6417de9cdfe66a8a4ae7604781"
server_key_iv: "67837abef05d64a739b5795ab624a514"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.529728 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:23.548897 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:23.552233 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:23.552433 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:23.553052 23045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:23.553930 23045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:23.554028 23045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.554096 23045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:23.554143 23045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.563054 23045 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:40015
I20260926 12:31:23.563073 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:40015 every 8 connection(s)
I20260926 12:31:23.563979 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:31:23.566715 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23045
I20260926 12:31:23.566881 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:31:23.567202 7564 external_mini_cluster.cc:1428] Setting key 5fc9564e3df4b6f5cc40a060cd4a6dab
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:31:23.577940 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:23.565770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:59169 (local address 127.7.99.62:44559)
0926 12:31:23.565922 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:23.565927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:23.566848 (+ 921us) server_negotiation.cc:407] Connection header received
0926 12:31:23.568443 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:23.568453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:23.568506 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:23.568583 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:23.569718 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.570257 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.571382 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.571625 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.571732 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:23.574186 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:23.574190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:23.574197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:23.574222 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:23.575650 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.576193 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.576199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.576204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.576289 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.576634 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.576637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.576639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.576924 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:23.577028 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:23.577547 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 12:31:23.577730 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:31:23.578874 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:23.566215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:59169)
0926 12:31:23.566666 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:23.566709 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:23.568250 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:23.568750 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:23.568757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:23.569111 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:23.569580 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:23.569594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.570381 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.570385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:23.571263 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:23.571272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.571769 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.571773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:23.571942 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:23.572571 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:23.572587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:23.574019 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:23.575782 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:23.575788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:23.575805 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:23.576069 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:23.576387 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:23.576390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:23.576392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:23.576534 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:23.577037 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:23.577046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:23.577301 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 12:31:23.577496 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 12:31:23.577858 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":148,"threads_started":1}
I20260926 12:31:23.580153 23174 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:23.580428 23174 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:23.580915 23174 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:23.582417 22983 ts_manager.cc:194] Registered new tserver with Master: a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015)
I20260926 12:31:23.583633 22983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:59169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:31:23.594805 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:23.712416 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:23.712656 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:23.712743 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:23.716826 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:23.716876 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:23.716975 23179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:31:23.722170 23179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:23.723167 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:23.724215 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:23.733471 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.733557 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.733495 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:23.733801 23179 server_base.cc:1034] running on GCE node
I20260926 12:31:23.734216 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:23.734869 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:23.736060 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790425883736030 us; error 35 us; skew 500 ppm
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:23.738718 23179 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:31:23.739769 23179 webserver.cc:460] Webserver started at http://127.7.99.2:43693/ using document root <none> and password file <none>
I20260926 12:31:23.740571 23179 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:23.740659 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:23.740880 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:23.743521 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "73eebbb49697477b87e4fd73fa382899"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "df24c0dad04cd4e2c0c0b49f432a21d9"
server_key_iv: "5fad14b0dfac365cbc5ee8118f601dbe"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.744092 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "73eebbb49697477b87e4fd73fa382899"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "df24c0dad04cd4e2c0c0b49f432a21d9"
server_key_iv: "5fad14b0dfac365cbc5ee8118f601dbe"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.747701 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:31:23.750144 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.751350 23179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:23.751487 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "73eebbb49697477b87e4fd73fa382899"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "df24c0dad04cd4e2c0c0b49f432a21d9"
server_key_iv: "5fad14b0dfac365cbc5ee8118f601dbe"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.751593 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:23.774219 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:23.777012 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:23.777196 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:23.777756 23179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:23.778568 23179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:23.778613 23179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.778678 23179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:23.778716 23179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.787266 23179 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:33779
I20260926 12:31:23.787292 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:33779 every 8 connection(s)
I20260926 12:31:23.788132 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:31:23.791473 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23179
I20260926 12:31:23.791610 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:31:23.791898 7564 external_mini_cluster.cc:1428] Setting key f50eeaf0fa66fec8eaea9eb569000bf3
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:31:23.801203 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:23.789813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:35029 (local address 127.7.99.62:44559)
0926 12:31:23.789935 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:23.789938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:23.790749 (+ 811us) server_negotiation.cc:407] Connection header received
0926 12:31:23.792286 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:23.792289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:23.792349 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:23.792456 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:23.793680 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.794175 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.795272 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.795497 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.795600 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:23.797832 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:23.797836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:23.797838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:23.797858 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:23.799214 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.799728 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.799733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.799735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.799810 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:23.800131 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:23.800134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:23.800136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:23.800409 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:23.800522 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:23.800968 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 12:31:23.801064 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:31:23.802023 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:23.790202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:35029)
0926 12:31:23.790619 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:23.790657 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:23.792034 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:23.792644 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:23.792653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:23.793112 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:23.793554 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:23.793567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.794286 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.794289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:23.795154 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:23.795164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:23.795632 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:23.795636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:23.795795 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:23.796350 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:23.796366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:23.797677 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:23.799347 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:23.799352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:23.799366 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:23.799630 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:23.799897 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:23.799902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:23.799904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:23.800034 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:23.800537 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:23.800542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:23.800755 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 12:31:23.800912 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 12:31:23.801155 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
I20260926 12:31:23.803121 23308 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:23.803365 23308 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:23.803825 23308 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:23.804931 22983 ts_manager.cc:194] Registered new tserver with Master: 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
I20260926 12:31:23.805748 22983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:35029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:31:23.819413 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:23.928120 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:23.928438 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:23.928514 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:23.932713 23313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:23.932783 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:23.932884 23313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:31:23.938094 23313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:23.939080 23313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:23.940071 23313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:23.949164 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.949141 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:23.949148 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:23.949196 23313 server_base.cc:1034] running on GCE node
I20260926 12:31:23.949718 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:23.950342 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:23.951514 23313 hybrid_clock.cc:648] HybridClock initialized: now 1790425883951501 us; error 29 us; skew 500 ppm
Sep 26 12:31:23 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:23.954282 23313 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:31:23.955194 23313 webserver.cc:460] Webserver started at http://127.7.99.3:46577/ using document root <none> and password file <none>
I20260926 12:31:23.955809 23313 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:23.955906 23313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:23.956107 23313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:31:23.958060 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "d96d825073e746b29d10361bd7d55d32"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "6f718cf04c815fd4d907da889a7ab169"
server_key_iv: "d99224f461429327f0937e95459309d0"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.958617 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "d96d825073e746b29d10361bd7d55d32"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "6f718cf04c815fd4d907da889a7ab169"
server_key_iv: "d99224f461429327f0937e95459309d0"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.962282 23313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:31:23.964725 23328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:23.965966 23313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:31:23.966161 23313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "d96d825073e746b29d10361bd7d55d32"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "6f718cf04c815fd4d907da889a7ab169"
server_key_iv: "d99224f461429327f0937e95459309d0"
server_key_version: "encryptionkey@0"
I20260926 12:31:23.966286 23313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:24.000770 23313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:24.003878 23313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:24.004066 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:24.004660 23313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:24.005458 23313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:24.005502 23313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:24.005564 23313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:24.005604 23313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:24.013783 23313 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33543
I20260926 12:31:24.013806 23441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33543 every 8 connection(s)
I20260926 12:31:24.014611 23313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:31:24.015165 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23313
I20260926 12:31:24.015291 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:31:24.015602 7564 external_mini_cluster.cc:1428] Setting key 455ba6da66ab75fef32df0a2b0509b43
Sep 26 12:31:24 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425883, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:24.027590 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.016400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:51609 (local address 127.7.99.62:44559)
0926 12:31:24.016522 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.016527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.017572 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 12:31:24.018892 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.018895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.018955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.019030 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:24.020063 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.020745 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.021798 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.022002 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.022078 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.024354 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:24.024358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:24.024360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:24.024386 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:24.025740 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:24.026167 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:24.026170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:24.026171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:24.026243 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:24.026553 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:24.026557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:24.026558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:24.026774 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:24.026878 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.027344 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.027455 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 12:31:24.028401 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.016932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:51609)
0926 12:31:24.017395 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.017463 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.018689 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.019152 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.019159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.019497 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:24.019942 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.019957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.020848 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.020850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.021695 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.021704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.022113 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.022116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.022248 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.022794 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:24.022809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:24.024123 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:24.025835 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:24.025839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:24.025850 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:24.026086 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:24.026315 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:24.026317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:24.026319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:24.026437 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:24.026881 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:24.026886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:24.027097 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.027239 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.027478 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260926 12:31:24.029500 23442 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:24.029770 23442 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:24.030329 23442 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:24.031318 22983 ts_manager.cc:194] Registered new tserver with Master: d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543)
I20260926 12:31:24.031852 22983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:51609
I20260926 12:31:24.041273 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:31:24.050177 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.043193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.7.99.62:44559)
0926 12:31:24.043305 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.043310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.043451 (+ 141us) server_negotiation.cc:407] Connection header received
0926 12:31:24.043556 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.043558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.043596 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.043664 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:24.044379 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.044888 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.045679 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.045893 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.045987 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.047264 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:24.047267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:24.047269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:24.047289 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:24.048574 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:24.048952 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:24.048954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:24.048956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:24.049032 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:24.049348 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:24.049351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:24.049353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:24.049608 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:24.049768 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.049931 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.050035 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:31:24.052961 22983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:31:24.055977 22983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:31:24.069948 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.064821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.7.99.2:33779)
0926 12:31:24.065463 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.065474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.065496 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:31:24.065580 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.065585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.065764 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.065928 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:24.066139 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.066943 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.068429 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.069225 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.069488 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.069631 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.069688 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.069787 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":128,"threads_started":1}
I20260926 12:31:24.070005 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.064189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.7.99.1:40015)
0926 12:31:24.065099 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.065115 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.065138 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:31:24.065220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.065228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.065489 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.066015 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:24.066161 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.067335 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.068676 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.069207 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.069325 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.069548 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.069727 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.069820 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":128,"threads_started":1}
I20260926 12:31:24.069996 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.064008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.0.0.1:41826)
0926 12:31:24.064502 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.064546 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.064670 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.065683 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.065690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.065715 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:24.065857 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.065863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.067513 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.067516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.068562 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.068572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.069325 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.069328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.069530 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.069554 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.069697 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.069814 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":108,"thread_start_us":71,"threads_started":1}
I20260926 12:31:24.070015 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.064183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:33779 (local address 127.0.0.1:34896)
0926 12:31:24.064670 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.065107 (+ 437us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.065196 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.065899 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.065902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.065918 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:24.066016 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.066022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.067071 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.067076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.068312 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.068322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.069367 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.069371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.069506 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.069551 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.069659 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.069799 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260926 12:31:24.071838 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.064727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33543 (local address 127.0.0.1:59368)
0926 12:31:24.065482 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.065496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.065584 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.066092 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.066094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.066112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:24.066187 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.066192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.067534 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.067537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.070112 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.070126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.071441 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.071456 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.071540 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.071562 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.071639 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.071701 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":107,"thread_start_us":89,"threads_started":1}
I20260926 12:31:24.071911 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.065128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.7.99.3:33543)
0926 12:31:24.065572 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.065585 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.065610 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:31:24.065685 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.065690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.065868 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.066013 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:24.066298 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.067401 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.070231 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.071159 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.071255 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.071382 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.071686 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.071777 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":125,"threads_started":1}
I20260926 12:31:24.073347 23108 tablet_service.cc:1467] Processing CreateTablet for tablet 3b779cc1051f4877938e4d10bc838f25 (DEFAULT_TABLE table=test-table [id=703cf18aef0347fc81b1c2caf1f69598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:24.073593 23242 tablet_service.cc:1467] Processing CreateTablet for tablet 3b779cc1051f4877938e4d10bc838f25 (DEFAULT_TABLE table=test-table [id=703cf18aef0347fc81b1c2caf1f69598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:24.074457 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b779cc1051f4877938e4d10bc838f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:24.074460 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b779cc1051f4877938e4d10bc838f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:24.074695 23376 tablet_service.cc:1467] Processing CreateTablet for tablet 3b779cc1051f4877938e4d10bc838f25 (DEFAULT_TABLE table=test-table [id=703cf18aef0347fc81b1c2caf1f69598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:24.075577 23376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b779cc1051f4877938e4d10bc838f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:24.080116 23459 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Bootstrap starting.
I20260926 12:31:24.081841 23460 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap starting.
I20260926 12:31:24.082520 23461 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Bootstrap starting.
I20260926 12:31:24.082767 23459 tablet_bootstrap.cc:654] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:24.083647 23459 log.cc:824] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:24.084084 23460 tablet_bootstrap.cc:654] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:24.084831 23461 tablet_bootstrap.cc:654] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:24.085125 23460 log.cc:824] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:24.085861 23459 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: No bootstrap required, opened a new log
I20260926 12:31:24.086054 23459 ts_tablet_manager.cc:1397] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:31:24.086026 23461 log.cc:824] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:24.087193 23460 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: No bootstrap required, opened a new log
I20260926 12:31:24.087443 23460 ts_tablet_manager.cc:1397] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:31:24.087917 23461 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: No bootstrap required, opened a new log
I20260926 12:31:24.088109 23461 ts_tablet_manager.cc:1397] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:31:24.089810 23459 raft_consensus.cc:348] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.090188 23459 raft_consensus.cc:374] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:24.090287 23459 raft_consensus.cc:729] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73eebbb49697477b87e4fd73fa382899, State: Initialized, Role: FOLLOWER
I20260926 12:31:24.090824 23459 consensus_queue.cc:260] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.091516 23308 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
I20260926 12:31:24.091782 23459 ts_tablet_manager.cc:1428] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:31:24.091820 23461 raft_consensus.cc:348] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.092082 23461 raft_consensus.cc:374] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:24.092180 23461 raft_consensus.cc:729] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96d825073e746b29d10361bd7d55d32, State: Initialized, Role: FOLLOWER
I20260926 12:31:24.092607 23461 consensus_queue.cc:260] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.092527 23460 raft_consensus.cc:348] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.092746 23460 raft_consensus.cc:374] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:24.092828 23460 raft_consensus.cc:729] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d79df178a64f85b88024b1c3d71f66, State: Initialized, Role: FOLLOWER
I20260926 12:31:24.093263 23442 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
I20260926 12:31:24.093240 23460 consensus_queue.cc:260] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.093559 23461 ts_tablet_manager.cc:1428] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260926 12:31:24.094043 23174 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
I20260926 12:31:24.094110 23460 ts_tablet_manager.cc:1428] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
W20260926 12:31:24.265575 23443 tablet.cc:2369] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:31:24.289266 23309 tablet.cc:2369] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:31:24.315327 23175 tablet.cc:2369] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:24.393934 23465 raft_consensus.cc:482] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:24.394200 23465 raft_consensus.cc:504] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.395394 23465 leader_election.cc:288] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015), d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543)
I20260926 12:31:24.399317 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.395858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:54843 (local address 127.7.99.3:33543)
0926 12:31:24.395991 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.395995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.396132 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:31:24.396334 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.396337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.396390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.396472 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:24.396735 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.397237 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.398325 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.398762 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.398890 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.398955 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.399124 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.399192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:31:24.399343 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.395697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33543 (local address 127.7.99.2:54843)
0926 12:31:24.396025 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.396041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.396208 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.396485 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.396488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.396506 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:24.396633 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.396639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.397353 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.397356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.398209 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.398218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.398897 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.398900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.398996 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.399019 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.399109 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.399180 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260926 12:31:24.399495 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.395817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:60481 (local address 127.7.99.1:40015)
0926 12:31:24.395958 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.395963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.396411 (+ 448us) server_negotiation.cc:407] Connection header received
0926 12:31:24.396540 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.396543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.396593 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.396684 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:24.396972 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.397477 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.398518 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.398919 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.399030 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.399092 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.399268 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.399345 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:31:24.399626 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.395697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.7.99.2:60481)
0926 12:31:24.396285 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.396296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:24.396430 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:24.396696 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:24.396699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:24.396714 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:24.396858 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.396864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.397585 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.397589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.398402 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:24.398410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.399037 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.399040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:24.399155 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.399172 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:31:24.399254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:31:24.399320 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":319,"thread_start_us":97,"threads_started":1}
I20260926 12:31:24.399976 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b779cc1051f4877938e4d10bc838f25" candidate_uuid: "73eebbb49697477b87e4fd73fa382899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96d825073e746b29d10361bd7d55d32" is_pre_election: true
I20260926 12:31:24.400259 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b779cc1051f4877938e4d10bc838f25" candidate_uuid: "73eebbb49697477b87e4fd73fa382899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d79df178a64f85b88024b1c3d71f66" is_pre_election: true
I20260926 12:31:24.400341 23396 raft_consensus.cc:2454] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73eebbb49697477b87e4fd73fa382899 in term 0.
I20260926 12:31:24.400561 23128 raft_consensus.cc:2454] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73eebbb49697477b87e4fd73fa382899 in term 0.
I20260926 12:31:24.400859 23196 leader_election.cc:302] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32; no voters:
I20260926 12:31:24.401147 23465 raft_consensus.cc:2781] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:31:24.401249 23465 raft_consensus.cc:482] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:31:24.401302 23465 raft_consensus.cc:3037] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:24.402420 23465 raft_consensus.cc:504] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.402763 23465 leader_election.cc:288] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [CANDIDATE]: Term 1 election: Requested vote from peers a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015), d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543)
I20260926 12:31:24.403039 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b779cc1051f4877938e4d10bc838f25" candidate_uuid: "73eebbb49697477b87e4fd73fa382899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d79df178a64f85b88024b1c3d71f66"
I20260926 12:31:24.403111 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b779cc1051f4877938e4d10bc838f25" candidate_uuid: "73eebbb49697477b87e4fd73fa382899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96d825073e746b29d10361bd7d55d32"
I20260926 12:31:24.403193 23128 raft_consensus.cc:3037] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:24.403265 23396 raft_consensus.cc:3037] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:24.404434 23396 raft_consensus.cc:2454] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73eebbb49697477b87e4fd73fa382899 in term 1.
I20260926 12:31:24.404438 23128 raft_consensus.cc:2454] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73eebbb49697477b87e4fd73fa382899 in term 1.
I20260926 12:31:24.404726 23198 leader_election.cc:302] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73eebbb49697477b87e4fd73fa382899, a3d79df178a64f85b88024b1c3d71f66; no voters:
I20260926 12:31:24.404918 23465 raft_consensus.cc:2781] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:24.405133 23465 raft_consensus.cc:686] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 LEADER]: Becoming Leader. State: Replica: 73eebbb49697477b87e4fd73fa382899, State: Running, Role: LEADER
I20260926 12:31:24.405462 23465 consensus_queue.cc:237] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:24.408740 22981 catalog_manager.cc:5495] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73eebbb49697477b87e4fd73fa382899 (127.7.99.2). New cstate: current_term: 1 leader_uuid: "73eebbb49697477b87e4fd73fa382899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260926 12:31:24.438160 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.434382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.7.99.2:33779)
0926 12:31:24.434502 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:24.434506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:24.434519 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:31:24.434678 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:24.434680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:24.434739 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:24.434834 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:24.435098 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.435647 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.436703 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:24.436963 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:24.437046 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:24.437377 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:24.437836 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:31:24.437929 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:24.437995 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 12:31:24.438043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:31:24.444216 23128 raft_consensus.cc:1261] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Refusing update from remote peer 73eebbb49697477b87e4fd73fa382899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:24.444239 23396 raft_consensus.cc:1261] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Refusing update from remote peer 73eebbb49697477b87e4fd73fa382899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:24.444921 23470 consensus_queue.cc:1035] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:24.445091 23465 consensus_queue.cc:1035] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:24.461234 22982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:24.461374 22982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:24.463572 22982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Sending DeleteTablet for 3 replicas of tablet 3b779cc1051f4877938e4d10bc838f25
I20260926 12:31:24.464254 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b779cc1051f4877938e4d10bc838f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:24 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:41826
I20260926 12:31:24.464381 23242 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b779cc1051f4877938e4d10bc838f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:24 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:34896
I20260926 12:31:24.464749 23483 tablet_replica.cc:330] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: stopping tablet replica
I20260926 12:31:24.464745 23376 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b779cc1051f4877938e4d10bc838f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:24 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 12:31:24.464975 23483 raft_consensus.cc:2229] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:24.465159 23485 tablet_replica.cc:330] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: stopping tablet replica
I20260926 12:31:24.465282 23483 raft_consensus.cc:2258] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:24.465453 23485 raft_consensus.cc:2229] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:24.465571 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23045
I20260926 12:31:24.465694 23485 raft_consensus.cc:2258] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:24.467820 23485 ts_tablet_manager.cc:1905] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:24.470293 23485 ts_tablet_manager.cc:1918] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:24.470395 23485 log.cc:1196] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/3b779cc1051f4877938e4d10bc838f25
I20260926 12:31:24.470717 23485 ts_tablet_manager.cc:1939] T 3b779cc1051f4877938e4d10bc838f25 P d96d825073e746b29d10361bd7d55d32: Deleting consensus metadata
I20260926 12:31:24.471782 22968 catalog_manager.cc:4841] TS d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): tablet 3b779cc1051f4877938e4d10bc838f25 (table test-table [id=703cf18aef0347fc81b1c2caf1f69598]) successfully deleted
I20260926 12:31:24.472347 23484 tablet_replica.cc:330] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: stopping tablet replica
I20260926 12:31:24.472636 23484 raft_consensus.cc:2229] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:31:24.473011 23484 raft_consensus.cc:2258] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:31:24.473567 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:40015 (error 108)
I20260926 12:31:24.474548 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.474268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.0.0.1:41830)
0926 12:31:24.474397 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.474465 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:31:24.474625 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23179
I20260926 12:31:24.474740 23484 ts_tablet_manager.cc:1905] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:31:24.476126 22970 catalog_manager.cc:4584] TS a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b779cc1051f4877938e4d10bc838f25: Network error: Client connection negotiation failed: client connection to 127.7.99.1:40015: connect: Connection refused (error 111)
I20260926 12:31:24.480711 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23313
I20260926 12:31:24.480837 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:24.480594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:33779 (local address 127.0.0.1:34908)
0926 12:31:24.480691 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:24.480736 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 12:31:24.486027 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 22951
I20260926 12:31:44.493626 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--webserver_interface=127.7.99.62
--webserver_port=40647
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:31:44.605924 23487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:44.606209 23487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:44.606324 23487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:44.610457 23487 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:31:44.610523 23487 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:31:44.610543 23487 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:31:44.610560 23487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:31:44.610620 23487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:44.610662 23487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:44.610702 23487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:31:44.610743 23487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:31:44.616254 23487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:44.617393 23487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:44.618467 23487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:44.631705 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:44.631705 23494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:44.631767 23493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:44.631973 23487 server_base.cc:1034] running on GCE node
I20260926 12:31:44.632620 23487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:44.633841 23487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:44.635040 23487 hybrid_clock.cc:648] HybridClock initialized: now 1790425904635020 us; error 31 us; skew 500 ppm
Sep 26 12:31:44 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425904, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:44.641959 23487 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:31:44.643061 23487 webserver.cc:460] Webserver started at http://127.7.99.62:40647/ using document root <none> and password file <none>
I20260926 12:31:44.643663 23487 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:44.643767 23487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:44.647784 23487 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 12:31:44.649953 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:44.651130 23487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:31:44.651265 23487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f4749c34cf624a089da3e339995b371f"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "995d35ffcc76ada1647d6d77bf8cfac0"
server_key_iv: "852444c6e5a13f12319fe1ad3ede1f00"
server_key_version: "encryptionkey@0"
I20260926 12:31:44.651855 23487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:44.669663 23487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:44.672632 23487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:44.672851 23487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:44.680092 23487 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44559
I20260926 12:31:44.680447 23555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44559 every 8 connection(s)
I20260926 12:31:44.681234 23487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:31:44.685545 23556 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:31:44.687219 23556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap starting.
I20260926 12:31:44.690763 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23487
I20260926 12:31:44.691248 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:40015
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=37679
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:44.705608 23556 log.cc:824] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:44.718142 23556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:44.718602 23556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap complete.
I20260926 12:31:44.722800 23556 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:44.723199 23556 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4749c34cf624a089da3e339995b371f, State: Initialized, Role: FOLLOWER
I20260926 12:31:44.723716 23556 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:44.723867 23556 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:31:44.723949 23556 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:31:44.724069 23556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:31:44.726079 23556 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:44.726440 23556 leader_election.cc:302] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4749c34cf624a089da3e339995b371f; no voters:
I20260926 12:31:44.726787 23556 leader_election.cc:288] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:31:44.726884 23560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:31:44.727142 23560 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 LEADER]: Becoming Leader. State: Replica: f4749c34cf624a089da3e339995b371f, State: Running, Role: LEADER
I20260926 12:31:44.727519 23560 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:31:44.728099 23556 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:31:44.728552 23560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4749c34cf624a089da3e339995b371f. Latest consensus state: current_term: 2 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:31:44.728765 23560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:44.728914 23561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:31:44.729046 23561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
I20260926 12:31:44.729318 23565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:31:44.734292 23565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:31:44.735036 23565 catalog_manager.cc:1259] Loaded cluster ID: 620d2bfe0fc24bb4ac1b88b996771247
I20260926 12:31:44.735123 23565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:31:44.737304 23565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:31:44.737926 23565 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Loaded TSK: 0
I20260926 12:31:44.752929 23565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Generated new TSK 1
I20260926 12:31:44.753216 23565 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:31:44.819977 23558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:44.820284 23558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:44.820369 23558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:44.824411 23558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:44.824467 23558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:44.824553 23558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:31:44.830339 23558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=37679
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:44.831362 23558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:44.832415 23558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:44.842259 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:44.842312 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:44.842555 23558 server_base.cc:1034] running on GCE node
W20260926 12:31:44.842588 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:44.843127 23558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:44.843873 23558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:44.845057 23558 hybrid_clock.cc:648] HybridClock initialized: now 1790425904845020 us; error 62 us; skew 500 ppm
Sep 26 12:31:44 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425904, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:44.847930 23558 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:31:44.848897 23558 webserver.cc:460] Webserver started at http://127.7.99.1:37679/ using document root <none> and password file <none>
I20260926 12:31:44.849488 23558 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:44.849560 23558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:44.853487 23558 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 12:31:44.855813 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:44.857136 23558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:31:44.857261 23558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "a3d79df178a64f85b88024b1c3d71f66"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "75e37c6417de9cdfe66a8a4ae7604781"
server_key_iv: "67837abef05d64a739b5795ab624a514"
server_key_version: "encryptionkey@0"
I20260926 12:31:44.857795 23558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:44.876089 23558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:44.879340 23558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:44.879562 23558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:44.880245 23558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:44.881420 23600 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:31:44.885560 23558 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:31:44.885627 23558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 12:31:44.885676 23558 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:31:44.887347 23558 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:31:44.887418 23558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:31:44.887676 23600 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap starting.
I20260926 12:31:44.898473 23558 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:40015
I20260926 12:31:44.898553 23707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:40015 every 8 connection(s)
I20260926 12:31:44.899765 23558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:31:44.908496 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23558
I20260926 12:31:44.909042 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:33779
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=43693
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:44.909170 23600 log.cc:824] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:31:44 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425904, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:44.917754 23600 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:44.918215 23600 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap complete.
I20260926 12:31:44.919512 23600 ts_tablet_manager.cc:1397] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20260926 12:31:44.920529 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:44.902033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:56457 (local address 127.7.99.62:44559)
0926 12:31:44.902393 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:44.902425 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:44.903710 (+ 1285us) server_negotiation.cc:407] Connection header received
0926 12:31:44.905183 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:44.905195 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:44.905558 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:44.905860 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:44.907490 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.908765 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.910205 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.910955 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.911138 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:44.914312 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:44.914318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:44.914333 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:44.914365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:44.916588 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:44.917327 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:44.917332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:44.917337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:44.917486 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:44.917955 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:44.917958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:44.917960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:44.918434 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:44.918573 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:44.919295 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 12:31:44.919606 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":129,"threads_started":1}
I20260926 12:31:44.920902 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:44.902046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:56457)
0926 12:31:44.903332 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:44.903390 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:44.904722 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:44.906030 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:44.906039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:44.906559 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:44.907304 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:44.907323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.908929 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.908932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:44.910067 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:44.910077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.911118 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.911123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:44.911306 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:44.912000 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:44.912019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:44.914084 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:44.916721 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:44.916728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:44.916745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:44.917188 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:44.917599 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:44.917604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:44.917610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:44.917804 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:44.918568 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:44.918578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:44.918915 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 12:31:44.919169 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 12:31:44.919553 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":147,"threads_started":1}
I20260926 12:31:44.922415 23600 raft_consensus.cc:348] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:44.922834 23600 raft_consensus.cc:729] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d79df178a64f85b88024b1c3d71f66, State: Initialized, Role: FOLLOWER
I20260926 12:31:44.923344 23600 consensus_queue.cc:260] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:44.923483 23708 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:44.923756 23708 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:44.924330 23708 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:44.924355 23600 ts_tablet_manager.cc:1428] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:31:44.926720 23520 ts_manager.cc:194] Registered new tserver with Master: a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015)
I20260926 12:31:44.930064 23520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:56457
I20260926 12:31:44.932382 23708 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
I20260926 12:31:44.936088 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:44.929708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.0.0.1:46020)
0926 12:31:44.930701 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:44.930866 (+ 165us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:44.930991 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:44.932665 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:44.932671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:44.932696 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:44.932820 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:44.932829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.933815 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.933819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:44.934690 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:44.934699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.935687 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.935689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:44.935765 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:44.935802 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 12:31:44.935881 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:31:44.935964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":130,"threads_started":1}
I20260926 12:31:44.936159 23716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:44.930509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.7.99.1:40015)
0926 12:31:44.930954 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:44.930965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:44.930987 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:31:44.932309 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:44.932314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:44.932495 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:44.932652 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:44.932925 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.933615 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.934813 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:44.935569 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:44.935678 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:44.935797 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:44.935928 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 12:31:44.936038 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":110,"threads_started":1}
I20260926 12:31:44.937182 23642 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b779cc1051f4877938e4d10bc838f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:24 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:46020
I20260926 12:31:44.938020 23717 tablet_replica.cc:330] stopping tablet replica
I20260926 12:31:44.938326 23717 raft_consensus.cc:2229] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:44.938575 23717 raft_consensus.cc:2258] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:44.939653 23717 ts_tablet_manager.cc:1905] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:44.944152 23717 ts_tablet_manager.cc:1918] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:44.944275 23717 log.cc:1196] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal/wals/3b779cc1051f4877938e4d10bc838f25
I20260926 12:31:44.944542 23717 ts_tablet_manager.cc:1939] T 3b779cc1051f4877938e4d10bc838f25 P a3d79df178a64f85b88024b1c3d71f66: Deleting consensus metadata
I20260926 12:31:44.946774 23507 catalog_manager.cc:4841] TS a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015): tablet 3b779cc1051f4877938e4d10bc838f25 (table test-table [id=703cf18aef0347fc81b1c2caf1f69598]) successfully deleted
W20260926 12:31:45.028661 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:45.028892 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:45.028975 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:45.033135 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:45.033188 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:45.033242 23713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:31:45.038540 23713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:33779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=43693
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:45.039429 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:45.040434 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:31:45.050035 23725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:45.050035 23722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:45.050032 23723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:45.050163 23713 server_base.cc:1034] running on GCE node
I20260926 12:31:45.050742 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:45.051366 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:45.052543 23713 hybrid_clock.cc:648] HybridClock initialized: now 1790425905052516 us; error 48 us; skew 500 ppm
Sep 26 12:31:45 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425905, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:45.055167 23713 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:31:45.056058 23713 webserver.cc:460] Webserver started at http://127.7.99.2:43693/ using document root <none> and password file <none>
I20260926 12:31:45.056672 23713 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:45.056772 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:45.061030 23713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:31:45.063347 23732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:45.064653 23713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:31:45.064781 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "73eebbb49697477b87e4fd73fa382899"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "df24c0dad04cd4e2c0c0b49f432a21d9"
server_key_iv: "5fad14b0dfac365cbc5ee8118f601dbe"
server_key_version: "encryptionkey@0"
I20260926 12:31:45.065312 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:45.080917 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:45.083835 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:45.084033 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:45.084653 23713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:45.085704 23739 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:31:45.089706 23713 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:31:45.089784 23713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 12:31:45.089867 23713 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:31:45.091467 23713 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:31:45.091533 23713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:31:45.091799 23739 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Bootstrap starting.
I20260926 12:31:45.097769 23739 log.cc:824] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:45.101464 23713 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:33779
I20260926 12:31:45.101537 23847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:33779 every 8 connection(s)
I20260926 12:31:45.102294 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:31:45.106189 23739 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:31:45.106407 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23713
I20260926 12:31:45.106630 23739 tablet_bootstrap.cc:492] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Bootstrap complete.
I20260926 12:31:45.106777 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:33543
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=46577
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:31:45.106956 23739 ts_tablet_manager.cc:1397] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 12:31:45.110574 23739 raft_consensus.cc:348] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
I20260926 12:31:45.111215 23739 raft_consensus.cc:729] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73eebbb49697477b87e4fd73fa382899, State: Initialized, Role: FOLLOWER
I20260926 12:31:45.111760 23739 consensus_queue.cc:260] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } } peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } }
Sep 26 12:31:45 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425905, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:45.112917 23739 ts_tablet_manager.cc:1428] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 12:31:45.116442 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.104151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:58747 (local address 127.7.99.62:44559)
0926 12:31:45.104309 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.104314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.104899 (+ 585us) server_negotiation.cc:407] Connection header received
0926 12:31:45.106082 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.106085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.106166 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.106267 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:45.107728 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.108447 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.109541 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.109777 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.109890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.112841 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:45.112845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:45.112848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:45.112881 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:45.114419 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.114925 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.114931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.114932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.115022 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.115359 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.115363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.115364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.115546 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:45.115653 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.116116 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.116272 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 12:31:45.117257 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.104304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:58747)
0926 12:31:45.104749 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.104801 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.105857 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.106428 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.106440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.106920 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:45.107549 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.107569 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.108597 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.108600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.109415 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.109423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.109886 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.109890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.110085 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.111017 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:45.111032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:45.112685 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:45.114552 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.114557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.114570 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.114823 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.115123 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.115126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.115127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.115260 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.115650 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:45.115655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:45.115886 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.116060 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.116383 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260926 12:31:45.118541 23848 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:45.118770 23848 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:45.119233 23848 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:45.120958 23520 ts_manager.cc:194] Registered new tserver with Master: 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
I20260926 12:31:45.122241 23520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:58747
I20260926 12:31:45.124459 23848 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
Sep 26 12:31:45 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425904, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.2@KRBTEST.COM
I20260926 12:31:45.132342 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.121923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:33779 (local address 127.0.0.1:59260)
0926 12:31:45.122146 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.122161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.122243 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.122898 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.122902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.123357 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:45.123460 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.123466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.124738 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.124741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.125597 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.125607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.126398 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.126410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.126478 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.127057 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:45.127068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:45.128479 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:45.130722 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.130731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.130736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.130976 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.131277 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.131279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.131280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.131354 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.131827 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:45.131832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:45.131933 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.132030 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.132140 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 12:31:45.132496 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.121983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.7.99.2:33779)
0926 12:31:45.122444 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.122455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.122497 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:31:45.122566 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.122571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.122761 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.122907 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:45.123588 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.124615 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.125736 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.126297 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.126420 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.128633 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:45.128644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:45.128658 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:45.128719 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:45.130625 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.131083 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.131089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.131091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.131183 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.131434 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.131437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.131438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.131720 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:45.131846 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.132135 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.132325 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":360,"thread_start_us":114,"threads_started":1}
I20260926 12:31:45.133458 23781 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b779cc1051f4877938e4d10bc838f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:24 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:59260
I20260926 12:31:45.134330 23854 tablet_replica.cc:330] stopping tablet replica
I20260926 12:31:45.134621 23854 raft_consensus.cc:2229] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:45.134856 23854 raft_consensus.cc:2258] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:45.135764 23854 ts_tablet_manager.cc:1905] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:45.139575 23854 ts_tablet_manager.cc:1918] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:45.139652 23854 log.cc:1196] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/3b779cc1051f4877938e4d10bc838f25
I20260926 12:31:45.139940 23854 ts_tablet_manager.cc:1939] T 3b779cc1051f4877938e4d10bc838f25 P 73eebbb49697477b87e4fd73fa382899: Deleting consensus metadata
I20260926 12:31:45.141713 23505 catalog_manager.cc:4841] TS 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): tablet 3b779cc1051f4877938e4d10bc838f25 (table test-table [id=703cf18aef0347fc81b1c2caf1f69598]) successfully deleted
W20260926 12:31:45.225479 23851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:31:45.225889 23851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:31:45.225971 23851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:31:45.230378 23851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:31:45.230453 23851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:31:45.230569 23851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:31:45.235934 23851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:33543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=46577
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:31:45.237010 23851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:31:45.238137 23851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:31:45.247983 23851 server_base.cc:1034] running on GCE node
W20260926 12:31:45.248135 23860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:45.248027 23862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:31:45.248039 23859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:31:45.248620 23851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:31:45.249320 23851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:31:45.250511 23851 hybrid_clock.cc:648] HybridClock initialized: now 1790425905250499 us; error 37 us; skew 500 ppm
Sep 26 12:31:45 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425905, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:31:45.253549 23851 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:31:45.254612 23851 webserver.cc:460] Webserver started at http://127.7.99.3:46577/ using document root <none> and password file <none>
I20260926 12:31:45.255226 23851 fs_manager.cc:359] Metadata directory not provided
I20260926 12:31:45.255342 23851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:31:45.259318 23851 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:31:45.261500 23869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:45.262774 23851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:31:45.262907 23851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "d96d825073e746b29d10361bd7d55d32"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "6f718cf04c815fd4d907da889a7ab169"
server_key_iv: "d99224f461429327f0937e95459309d0"
server_key_version: "encryptionkey@0"
I20260926 12:31:45.263459 23851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:31:45.277503 23851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:31:45.280332 23851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:31:45.280532 23851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:31:45.281111 23851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:31:45.281917 23851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:31:45.281996 23851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:45.282071 23851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:31:45.282124 23851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:31:45.290812 23851 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33543
I20260926 12:31:45.290844 23982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33543 every 8 connection(s)
I20260926 12:31:45.291811 23851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:31:45.293723 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 23851
Sep 26 12:31:45 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425905, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:31:45.302775 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.294336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.7.99.62:44559)
0926 12:31:45.294656 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.294661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.294725 (+ 64us) server_negotiation.cc:407] Connection header received
0926 12:31:45.294882 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.294885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.294931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.295043 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:45.295754 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.296627 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.297496 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.297822 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.297897 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.299323 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:45.299327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:45.299330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:45.299356 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:45.300983 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.301450 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.301455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.301457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.301542 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.301826 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.301829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.301830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.301994 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:45.302143 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.302369 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.302541 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":254,"thread_start_us":78,"threads_started":1}
I20260926 12:31:45.306332 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.293547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:47261 (local address 127.7.99.62:44559)
0926 12:31:45.293713 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.293718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.294634 (+ 916us) server_negotiation.cc:407] Connection header received
0926 12:31:45.296208 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.296214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.296259 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.296323 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:45.297601 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.298234 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.299479 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.299687 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.299795 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.302344 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:45.302348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:45.302350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:45.302368 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:45.304249 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.304844 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.304849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.304851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.304930 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.305283 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.305287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.305289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.305451 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:45.305562 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.306062 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.306171 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":68}
I20260926 12:31:45.307103 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.293982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:47261)
0926 12:31:45.294492 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.294545 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.295954 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.296537 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.296546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.296958 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:31:45.297408 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.297420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.298365 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.298370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.299360 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.299370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.299845 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.299849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.300016 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.300656 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:31:45.300671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:31:45.302161 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:31:45.304362 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.304368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.304382 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.304743 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.305037 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:31:45.305040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:31:45.305042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:31:45.305186 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:31:45.305561 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:31:45.305569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:31:45.305809 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.305999 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.306310 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260926 12:31:45.308182 23983 heartbeater.cc:347] Connected to a master server at 127.7.99.62:44559
I20260926 12:31:45.308424 23983 heartbeater.cc:464] Registering TS with master...
I20260926 12:31:45.308882 23983 heartbeater.cc:511] Master 127.7.99.62:44559 requested a full tablet report, sending...
I20260926 12:31:45.309852 23520 ts_manager.cc:194] Registered new tserver with Master: d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543)
I20260926 12:31:45.310397 23520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:47261
I20260926 12:31:45.316236 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:31:45.319836 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.316865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.7.99.62:44559)
0926 12:31:45.316982 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.316985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.317146 (+ 161us) server_negotiation.cc:407] Connection header received
0926 12:31:45.317279 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.317281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.317367 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.317442 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:45.317637 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.318114 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.318886 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.319089 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.319203 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.319434 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:45.319558 (+ 124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 12:31:45.319745 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59632: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 12:31:45.319913 23710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59632: token expired
W20260926 12:31:45.320288 23445 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.99.62:44559)
I20260926 12:31:45.327227 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.320617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.7.99.62:44559)
0926 12:31:45.320726 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.320731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.320767 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:31:45.320903 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.320906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.320944 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.321041 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:31:45.321630 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.322102 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.322876 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.323101 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.323203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.324425 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:31:45.324429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:31:45.324432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:31:45.324461 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:31:45.325595 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.326016 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.326020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.326022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.326102 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:31:45.326370 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:31:45.326374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:31:45.326376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:31:45.326555 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:31:45.326709 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.326953 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.327073 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:31:45.329674 23520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:31:45.332967 23520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:31:45.339864 23781 tablet_service.cc:1467] Processing CreateTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 (DEFAULT_TABLE table=test-table [id=8b5fb8707f2a459ebedbe757663f56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:45.339864 23642 tablet_service.cc:1467] Processing CreateTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 (DEFAULT_TABLE table=test-table [id=8b5fb8707f2a459ebedbe757663f56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:45.340502 23781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fec2d2037545409aaf1b2420d3dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:45.340502 23642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fec2d2037545409aaf1b2420d3dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:45.343343 23600 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap starting.
I20260926 12:31:45.343894 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.339149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33543 (local address 127.0.0.1:46386)
0926 12:31:45.339278 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.339295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.339369 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.340319 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.340322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.340340 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:45.340471 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.340478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.341613 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.341618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.342601 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.342611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.343534 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.343538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.343622 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.343647 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.343697 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.343767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:31:45.344276 23600 tablet_bootstrap.cc:654] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:45.344565 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.339522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.7.99.3:33543)
0926 12:31:45.339818 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.339829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.339859 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:31:45.339921 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.339925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.340148 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.340356 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:45.340589 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.341477 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.342723 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.343402 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.344069 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.344264 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.344319 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.344415 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260926 12:31:45.345043 23739 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Bootstrap starting.
I20260926 12:31:45.345842 23600 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: No bootstrap required, opened a new log
I20260926 12:31:45.345933 23600 ts_tablet_manager.cc:1397] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 12:31:45.346047 23739 tablet_bootstrap.cc:654] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:45.346462 23600 raft_consensus.cc:348] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.346597 23600 raft_consensus.cc:374] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:45.346634 23600 raft_consensus.cc:729] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d79df178a64f85b88024b1c3d71f66, State: Initialized, Role: FOLLOWER
I20260926 12:31:45.346768 23600 consensus_queue.cc:260] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.347048 23600 ts_tablet_manager.cc:1428] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:31:45.347698 23739 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: No bootstrap required, opened a new log
I20260926 12:31:45.347782 23739 ts_tablet_manager.cc:1397] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:31:45.347703 23917 tablet_service.cc:1467] Processing CreateTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 (DEFAULT_TABLE table=test-table [id=8b5fb8707f2a459ebedbe757663f56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:31:45.348270 23739 raft_consensus.cc:348] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.348402 23739 raft_consensus.cc:374] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:45.348460 23739 raft_consensus.cc:729] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73eebbb49697477b87e4fd73fa382899, State: Initialized, Role: FOLLOWER
I20260926 12:31:45.348593 23739 consensus_queue.cc:260] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.348923 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fec2d2037545409aaf1b2420d3dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:31:45.348930 23739 ts_tablet_manager.cc:1428] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 12:31:45.353223 23849 tablet.cc:2369] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:45.355887 23995 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Bootstrap starting.
I20260926 12:31:45.357851 23995 tablet_bootstrap.cc:654] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Neither blocks nor log segments found. Creating new log.
I20260926 12:31:45.358875 23995 log.cc:824] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Log is configured to *not* fsync() on all Append() calls
I20260926 12:31:45.360668 23995 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: No bootstrap required, opened a new log
I20260926 12:31:45.360864 23995 ts_tablet_manager.cc:1397] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:31:45.363847 23995 raft_consensus.cc:348] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.364069 23995 raft_consensus.cc:374] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:31:45.364151 23995 raft_consensus.cc:729] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96d825073e746b29d10361bd7d55d32, State: Initialized, Role: FOLLOWER
I20260926 12:31:45.364684 23995 consensus_queue.cc:260] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.365324 23983 heartbeater.cc:503] Master 127.7.99.62:44559 was elected leader, sending a full tablet report...
I20260926 12:31:45.365653 23995 ts_tablet_manager.cc:1428] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260926 12:31:45.401197 23709 tablet.cc:2369] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:45.472504 23997 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:31:45.472780 23997 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.474047 23997 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
I20260926 12:31:45.479290 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.474482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:54971 (local address 127.7.99.2:33779)
0926 12:31:45.474610 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.474615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.474973 (+ 358us) server_negotiation.cc:407] Connection header received
0926 12:31:45.475058 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.475061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.475124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.475225 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:45.475512 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.476298 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478031 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.478714 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478809 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.478979 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.479040 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.479120 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:31:45.479271 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.474483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:41633 (local address 127.7.99.1:40015)
0926 12:31:45.474800 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.474808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.474833 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:31:45.474886 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.474888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.474953 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.475067 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:31:45.475390 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.476262 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478036 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.478651 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478783 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.478864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.479063 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.479142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260926 12:31:45.479278 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.474361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.7.99.3:41633)
0926 12:31:45.474513 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.474532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.474653 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.475098 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.475103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.475122 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:45.475257 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.475263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.476407 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.476411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.477873 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.477887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.478798 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.478920 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.478946 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.479025 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.479103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":107}
I20260926 12:31:45.479290 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.474361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:33779 (local address 127.7.99.3:54971)
0926 12:31:45.474878 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.474890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.474981 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.475259 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:31:45.475261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:31:45.475275 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:31:45.475399 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.475404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.476407 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.476411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.477887 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:31:45.477894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.478832 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.478835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:31:45.478923 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.478956 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 12:31:45.479016 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:31:45.479103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":280,"thread_start_us":136,"threads_started":1}
I20260926 12:31:45.479954 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fec2d2037545409aaf1b2420d3dd06" candidate_uuid: "d96d825073e746b29d10361bd7d55d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d79df178a64f85b88024b1c3d71f66" is_pre_election: true
I20260926 12:31:45.479959 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fec2d2037545409aaf1b2420d3dd06" candidate_uuid: "d96d825073e746b29d10361bd7d55d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eebbb49697477b87e4fd73fa382899" is_pre_election: true
I20260926 12:31:45.480315 23662 raft_consensus.cc:2454] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d96d825073e746b29d10361bd7d55d32 in term 0.
I20260926 12:31:45.480316 23800 raft_consensus.cc:2454] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d96d825073e746b29d10361bd7d55d32 in term 0.
I20260926 12:31:45.480834 23871 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73eebbb49697477b87e4fd73fa382899, a3d79df178a64f85b88024b1c3d71f66, d96d825073e746b29d10361bd7d55d32; no voters:
I20260926 12:31:45.481103 23997 raft_consensus.cc:2781] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:31:45.481197 23997 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:31:45.481258 23997 raft_consensus.cc:3037] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:45.482293 23997 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.482702 23997 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [CANDIDATE]: Term 1 election: Requested vote from peers a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
I20260926 12:31:45.483093 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fec2d2037545409aaf1b2420d3dd06" candidate_uuid: "d96d825073e746b29d10361bd7d55d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d79df178a64f85b88024b1c3d71f66"
I20260926 12:31:45.483099 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fec2d2037545409aaf1b2420d3dd06" candidate_uuid: "d96d825073e746b29d10361bd7d55d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73eebbb49697477b87e4fd73fa382899"
I20260926 12:31:45.483244 23800 raft_consensus.cc:3037] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:45.483235 23662 raft_consensus.cc:3037] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:31:45.484335 23662 raft_consensus.cc:2454] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d96d825073e746b29d10361bd7d55d32 in term 1.
I20260926 12:31:45.484338 23800 raft_consensus.cc:2454] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d96d825073e746b29d10361bd7d55d32 in term 1.
I20260926 12:31:45.484645 23871 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32; no voters:
I20260926 12:31:45.484839 23997 raft_consensus.cc:2781] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:31:45.485080 23997 raft_consensus.cc:686] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 1 LEADER]: Becoming Leader. State: Replica: d96d825073e746b29d10361bd7d55d32, State: Running, Role: LEADER
I20260926 12:31:45.485461 23997 consensus_queue.cc:237] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:31:45.489042 23519 catalog_manager.cc:5495] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 reported cstate change: term changed from 0 to 1, leader changed from <none> to d96d825073e746b29d10361bd7d55d32 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "d96d825073e746b29d10361bd7d55d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } health_report { overall_health: UNKNOWN } } }
I20260926 12:31:45.535851 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:31:45.532285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.7.99.3:33543)
0926 12:31:45.532435 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:31:45.532440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:31:45.532455 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:31:45.532638 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:31:45.532641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:31:45.532702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:31:45.532819 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:31:45.533065 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.533573 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.534392 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:31:45.534613 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:31:45.534724 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:31:45.535035 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:31:45.535446 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:31:45.535584 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:31:45.535625 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 12:31:45.535680 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:31:45.541832 23662 raft_consensus.cc:1261] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Refusing update from remote peer d96d825073e746b29d10361bd7d55d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:31:45.541865 23800 raft_consensus.cc:1261] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Refusing update from remote peer d96d825073e746b29d10361bd7d55d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 12:31:45.542660 23984 tablet.cc:2369] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:31:45.542605 24000 consensus_queue.cc:1035] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:45.542824 23997 consensus_queue.cc:1035] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:31:45.548614 24003 mvcc.cc:205] Tried to move back new op lower bound from 7333584509091991552 to 7333584508872081408. Current Snapshot: MvccSnapshot[applied={T|T < 7333584509091991552}]
I20260926 12:31:45.557271 23520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:45.557451 23520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:31:45.559630 23520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Sending DeleteTablet for 3 replicas of tablet 63fec2d2037545409aaf1b2420d3dd06
I20260926 12:31:45.560377 23642 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:46020
I20260926 12:31:45.560335 23917 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:46386
I20260926 12:31:45.560811 24014 tablet_replica.cc:330] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: stopping tablet replica
I20260926 12:31:45.561024 23781 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fec2d2037545409aaf1b2420d3dd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:31:45 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:59260
I20260926 12:31:45.561146 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23558
I20260926 12:31:45.561242 23854 tablet_replica.cc:330] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: stopping tablet replica
I20260926 12:31:45.561373 23854 raft_consensus.cc:2229] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:31:45.561494 23854 raft_consensus.cc:2258] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:45.562639 24014 raft_consensus.cc:2229] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:31:45.562791 23854 ts_tablet_manager.cc:1905] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:31:45.562994 24014 raft_consensus.cc:2258] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:31:45.565088 23854 ts_tablet_manager.cc:1918] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:45.565194 23854 log.cc:1196] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/63fec2d2037545409aaf1b2420d3dd06
I20260926 12:31:45.565435 23854 ts_tablet_manager.cc:1939] T 63fec2d2037545409aaf1b2420d3dd06 P 73eebbb49697477b87e4fd73fa382899: Deleting consensus metadata
I20260926 12:31:45.566269 23505 catalog_manager.cc:4841] TS 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): tablet 63fec2d2037545409aaf1b2420d3dd06 (table test-table [id=8b5fb8707f2a459ebedbe757663f56a7]) successfully deleted
W20260926 12:31:45.567170 23507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:40015 (error 108)
W20260926 12:31:45.568715 23715 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:31:45.567932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:40015 (local address 127.0.0.1:44420)
0926 12:31:45.568057 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:31:45.568072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:31:45.568205 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:31:45.568600 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:40015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 12:31:45.568928 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23713
I20260926 12:31:45.570220 24014 ts_tablet_manager.cc:1905] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:31:45.570895 23507 catalog_manager.cc:4584] TS a3d79df178a64f85b88024b1c3d71f66 (127.7.99.1:40015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63fec2d2037545409aaf1b2420d3dd06: Network error: Client connection negotiation failed: client connection to 127.7.99.1:40015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:31:45.573264 24014 ts_tablet_manager.cc:1918] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:31:45.573367 24014 log.cc:1196] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/63fec2d2037545409aaf1b2420d3dd06
I20260926 12:31:45.573705 24014 ts_tablet_manager.cc:1939] T 63fec2d2037545409aaf1b2420d3dd06 P d96d825073e746b29d10361bd7d55d32: Deleting consensus metadata
I20260926 12:31:45.575253 23505 catalog_manager.cc:4841] TS d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): tablet 63fec2d2037545409aaf1b2420d3dd06 (table test-table [id=8b5fb8707f2a459ebedbe757663f56a7]) successfully deleted
I20260926 12:31:45.577149 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23851
I20260926 12:31:45.583184 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 23487
I20260926 12:32:05.589771 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--webserver_interface=127.7.99.62
--webserver_port=40647
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:05.698673 24016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:05.698945 24016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:05.698987 24016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:05.703078 24016 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:32:05.703138 24016 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:32:05.703154 24016 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:32:05.703167 24016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:05.703186 24016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:05.703255 24016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:05.703285 24016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:05.703316 24016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:05.708828 24016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:05.710021 24016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:05.711184 24016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:05.720549 24024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:05.720611 24016 server_base.cc:1034] running on GCE node
W20260926 12:32:05.720559 24022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:05.720558 24021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:05.721565 24016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:05.722756 24016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:05.723984 24016 hybrid_clock.cc:648] HybridClock initialized: now 1790425925723946 us; error 52 us; skew 500 ppm
Sep 26 12:32:05 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:05.733743 24016 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:05.734799 24016 webserver.cc:460] Webserver started at http://127.7.99.62:40647/ using document root <none> and password file <none>
I20260926 12:32:05.735368 24016 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:05.735432 24016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:05.739395 24016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:05.741637 24031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:05.743040 24016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:32:05.743162 24016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f4749c34cf624a089da3e339995b371f"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "995d35ffcc76ada1647d6d77bf8cfac0"
server_key_iv: "852444c6e5a13f12319fe1ad3ede1f00"
server_key_version: "encryptionkey@0"
I20260926 12:32:05.743706 24016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:05.767127 24016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:05.770586 24016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:05.770797 24016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:05.778273 24016 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44559
I20260926 12:32:05.778630 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44559 every 8 connection(s)
I20260926 12:32:05.779394 24016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:05.783643 24084 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:32:05.785272 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap starting.
I20260926 12:32:05.786795 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24016
I20260926 12:32:05.787238 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:40015
--local_ip_for_outbound_sockets=127.7.99.1
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=37679
--webserver_interface=127.7.99.1
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:32:05.803542 24084 log.cc:824] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:05.816002 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:32:05.816668 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Bootstrap complete.
I20260926 12:32:05.820789 24084 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:32:05.821163 24084 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4749c34cf624a089da3e339995b371f, State: Initialized, Role: FOLLOWER
I20260926 12:32:05.821676 24084 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:32:05.821791 24084 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:05.821882 24084 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:05.821985 24084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:32:05.823859 24084 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:32:05.824257 24084 leader_election.cc:302] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4749c34cf624a089da3e339995b371f; no voters:
I20260926 12:32:05.824604 24084 leader_election.cc:288] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:32:05.824697 24088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:32:05.824922 24088 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [term 3 LEADER]: Becoming Leader. State: Replica: f4749c34cf624a089da3e339995b371f, State: Running, Role: LEADER
I20260926 12:32:05.825212 24088 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } }
I20260926 12:32:05.825905 24084 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:05.826481 24089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:32:05.826601 24089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:05.826822 24089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4749c34cf624a089da3e339995b371f. Latest consensus state: current_term: 3 leader_uuid: "f4749c34cf624a089da3e339995b371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4749c34cf624a089da3e339995b371f" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44559 } } }
I20260926 12:32:05.826926 24089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:05.827574 24093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:05.832607 24093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:05.833269 24093 catalog_manager.cc:1259] Loaded cluster ID: 620d2bfe0fc24bb4ac1b88b996771247
I20260926 12:32:05.833330 24093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:05.835855 24093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:05.836601 24093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Loaded TSK: 0
I20260926 12:32:05.836696 24093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4749c34cf624a089da3e339995b371f: Loaded TSK: 1
F20260926 12:32:05.837440 24093 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa8b4bc99ad google::LogMessage::Fail() at ??:0
@ 0x7fa8b4bcc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa8b4bc94de google::LogMessage::Flush() at ??:0
@ 0x7fa8b4bcaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa8b788d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa8b7885f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa8bf52e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa8bf50899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa8bf509178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fa8bf54adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa8bf56982b std::function<>::operator()() at ??:0
@ 0x7fa8bf508b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa8bf509f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa8bf506579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa8bf54a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa8b6e1b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa8b6e1bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa8b6e1d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa8b6e0c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa8b84506db start_thread at ??:0
@ 0x7fa8b36c761f clone at ??:0
W20260926 12:32:05.938237 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:05.938630 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:05.938726 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:05.943557 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:05.943614 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:05.943708 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:05.949302 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=37679
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:05.950383 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:05.951434 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:05.961207 24131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:05.961117 24132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:05.961359 24086 server_base.cc:1034] running on GCE node
W20260926 12:32:05.961114 24134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:05.962014 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:05.962828 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:05.964031 24086 hybrid_clock.cc:648] HybridClock initialized: now 1790425925964001 us; error 56 us; skew 500 ppm
Sep 26 12:32:05 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:05.967376 24086 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:32:05.968359 24086 webserver.cc:460] Webserver started at http://127.7.99.1:37679/ using document root <none> and password file <none>
I20260926 12:32:05.968935 24086 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:05.968997 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:05.972904 24086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:32:05.975160 24141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:05.976536 24086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:32:05.976675 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "a3d79df178a64f85b88024b1c3d71f66"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "75e37c6417de9cdfe66a8a4ae7604781"
server_key_iv: "67837abef05d64a739b5795ab624a514"
server_key_version: "encryptionkey@0"
I20260926 12:32:05.977213 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:05.989480 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:05.992959 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:05.993170 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:05.993774 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:05.994899 24148 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:32:06.000130 24086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:32:06.000267 24086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 12:32:06.000352 24086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:32:06.002002 24086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:32:06.002076 24086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:32:06.002364 24148 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap starting.
I20260926 12:32:06.012714 24086 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:40015
I20260926 12:32:06.012945 24255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:40015 every 8 connection(s)
I20260926 12:32:06.013717 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:06.014382 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24086
I20260926 12:32:06.014907 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:33779
--local_ip_for_outbound_sockets=127.7.99.2
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=43693
--webserver_interface=127.7.99.2
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:32:06.025974 24148 log.cc:824] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:06.034859 24148 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:32:06.035321 24148 tablet_bootstrap.cc:492] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Bootstrap complete.
I20260926 12:32:06.036790 24148 ts_tablet_manager.cc:1397] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.009s
I20260926 12:32:06.040024 24148 raft_consensus.cc:348] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:06.040570 24148 raft_consensus.cc:729] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d79df178a64f85b88024b1c3d71f66, State: Initialized, Role: FOLLOWER
I20260926 12:32:06.041075 24148 consensus_queue.cc:260] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:06.042073 24148 ts_tablet_manager.cc:1428] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260926 12:32:06.065260 24259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:06.018576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:42749)
0926 12:32:06.019092 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.019135 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:06.019883 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:06.064329 (+ 44446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
W20260926 12:32:06.065791 24256 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44559 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:44559: Client connection negotiation failed: client connection to 127.7.99.62:44559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:32:06.066587 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.066283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:54431)
0926 12:32:06.066437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.066523 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:32:06.067307 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.067063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:50261)
0926 12:32:06.067184 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.067231 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:32:06.067504 24256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:32:06.129819 24258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:06.130112 24258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:06.130155 24258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:06.134142 24258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:06.134197 24258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:06.134305 24258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:06.139585 24258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:33779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=43693
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:06.140694 24258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:06.141816 24258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:06.151437 24269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:06.151408 24267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:06.151433 24266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:06.151486 24258 server_base.cc:1034] running on GCE node
I20260926 12:32:06.152009 24258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:06.152654 24258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:06.153815 24258 hybrid_clock.cc:648] HybridClock initialized: now 1790425926153787 us; error 40 us; skew 500 ppm
Sep 26 12:32:06 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425926, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:06.156451 24258 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:32:06.157228 24258 webserver.cc:460] Webserver started at http://127.7.99.2:43693/ using document root <none> and password file <none>
I20260926 12:32:06.157785 24258 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:06.157847 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:06.161507 24258 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:06.163492 24276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.164742 24258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:06.164865 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "73eebbb49697477b87e4fd73fa382899"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "df24c0dad04cd4e2c0c0b49f432a21d9"
server_key_iv: "5fad14b0dfac365cbc5ee8118f601dbe"
server_key_version: "encryptionkey@0"
I20260926 12:32:06.165380 24258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:06.176524 24258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:06.179749 24258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:06.179963 24258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:06.180616 24258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:06.181488 24258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:06.181535 24258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.181607 24258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:06.181641 24258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.190564 24258 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:33779
I20260926 12:32:06.190620 24389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:33779 every 8 connection(s)
I20260926 12:32:06.191633 24258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:06.194380 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24258
I20260926 12:32:06.194829 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:33543
--local_ip_for_outbound_sockets=127.7.99.3
--tserver_master_addrs=127.7.99.62:44559
--webserver_port=46577
--webserver_interface=127.7.99.3
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:32:06.195629 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.193759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:33665)
0926 12:32:06.194214 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.194318 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
W20260926 12:32:06.196400 24390 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44559 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:44559: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
I20260926 12:32:06.197007 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.196773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:60215)
0926 12:32:06.196885 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.196955 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 12:32:06.197703 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.197522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:56401)
0926 12:32:06.197618 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.197658 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 12:32:06.197947 24390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:32:06.304891 24393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:06.305122 24393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:06.305195 24393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:06.309145 24393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:06.309198 24393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:06.309296 24393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:06.314520 24393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:33543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=46577
--tserver_master_addrs=127.7.99.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:06.315490 24393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:06.316555 24393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:06.326149 24399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:06.326218 24398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:06.326260 24401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:06.326293 24393 server_base.cc:1034] running on GCE node
I20260926 12:32:06.326769 24393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:06.327322 24393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:06.328480 24393 hybrid_clock.cc:648] HybridClock initialized: now 1790425926328466 us; error 24 us; skew 500 ppm
Sep 26 12:32:06 dist-test-slave-3jw2 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425926, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:06.331226 24393 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:32:06.331996 24393 webserver.cc:460] Webserver started at http://127.7.99.3:46577/ using document root <none> and password file <none>
I20260926 12:32:06.332556 24393 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:06.332615 24393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:06.336140 24393 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:06.338155 24408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.339377 24393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:06.339505 24393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "d96d825073e746b29d10361bd7d55d32"
format_stamp: "Formatted at 2026-09-26 12:31:23 on dist-test-slave-3jw2"
server_key: "6f718cf04c815fd4d907da889a7ab169"
server_key_iv: "d99224f461429327f0937e95459309d0"
server_key_version: "encryptionkey@0"
I20260926 12:32:06.340057 24393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:06.351281 24393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:06.354518 24393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:06.354727 24393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:06.355337 24393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:06.356154 24393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:06.356290 24393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.356365 24393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:06.356402 24393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:06.365695 24393 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33543
I20260926 12:32:06.365854 24521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33543 every 8 connection(s)
I20260926 12:32:06.366781 24393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:06.371476 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24393
I20260926 12:32:06.372020 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.369726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:60749)
0926 12:32:06.370375 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.370514 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":172,"threads_started":1}
W20260926 12:32:06.372718 24522 heartbeater.cc:644] Failed to heartbeat to 127.7.99.62:44559 (0 consecutive failures): Network error: Failed to ping master at 127.7.99.62:44559: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
I20260926 12:32:06.373464 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.373202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:59411)
0926 12:32:06.373345 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.373396 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 12:32:06.374207 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:06.374013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:54085)
0926 12:32:06.374117 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:06.374159 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:32:06.374506 24522 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:32:07.068758 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.068110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:57437)
0926 12:32:07.068498 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:07.068563 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":118,"threads_started":1}
I20260926 12:32:07.198870 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.198476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:59097)
0926 12:32:07.198700 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:07.198754 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260926 12:32:07.255539 24528 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:07.255762 24528 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:07.257030 24528 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
Sep 26 12:32:07 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.2@KRBTEST.COM
Sep 26 12:32:07 dist-test-slave-3jw2 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.3@KRBTEST.COM
I20260926 12:32:07.271425 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.257771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:35991 (local address 127.7.99.2:33779)
0926 12:32:07.258054 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:07.258081 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:07.258111 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:32:07.258704 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:07.258709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:07.259002 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:07.259285 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:07.260460 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.261385 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.262568 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.263045 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.263219 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:07.266029 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:07.266034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:07.266045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:07.266070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:07.268196 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:07.268955 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:07.268961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:07.268967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:07.269136 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:07.269680 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:07.269684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:07.269686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:07.270116 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:07.270227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:07.270956 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 12:32:07.271232 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260926 12:32:07.271425 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.257409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51065 (local address 127.7.99.3:33543)
0926 12:32:07.257783 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:07.257799 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:07.257846 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:32:07.258546 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:07.258552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:07.258899 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:07.259206 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:07.260458 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.261406 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.262570 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.263046 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.263218 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:07.266338 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:07.266345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:07.266361 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:07.266397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:07.268196 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:07.268955 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:07.268961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:07.268967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:07.269126 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:07.269507 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:07.269509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:07.269510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:07.270046 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:07.270184 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:07.270956 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 12:32:07.271232 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":72,"threads_started":1}
I20260926 12:32:07.271464 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.257209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33543 (local address 127.7.99.1:51065)
0926 12:32:07.257363 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:07.257377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:07.257502 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:07.259245 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:07.259251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:07.259790 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:07.260298 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:07.260314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.261533 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.261539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:07.262444 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:07.262453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.263174 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.263177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:07.263383 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:07.264025 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:07.264045 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:07.266142 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:07.268318 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:07.268323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:07.268338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:07.268801 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:07.269260 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:07.269263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:07.269264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:07.269408 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:07.270227 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:07.270235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:07.270584 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 12:32:07.270826 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 12:32:07.271190 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":244,"spinlock_wait_cycles":22400}
I20260926 12:32:07.271466 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.257510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:33779 (local address 127.7.99.1:35991)
0926 12:32:07.257720 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:07.257733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:07.257832 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:07.259247 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:07.259253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:07.259788 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:07.260298 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:07.260313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.261540 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.261545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:07.262446 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:07.262454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:07.263169 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:07.263173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:07.263379 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:07.264098 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:07.264116 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:07.265781 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:07.268317 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:07.268332 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:07.268338 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:07.268833 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:07.269514 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:07.269516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:07.269517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:07.269591 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:07.270227 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:07.270235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:07.270584 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 12:32:07.270826 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 12:32:07.271248 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":251,"spinlock_wait_cycles":33280,"thread_start_us":62,"threads_started":1}
W20260926 12:32:07.274344 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
W20260926 12:32:07.274529 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
I20260926 12:32:07.274580 24143 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3d79df178a64f85b88024b1c3d71f66; no voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32
I20260926 12:32:07.275063 24528 raft_consensus.cc:2726] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:32:07.375388 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:07.375001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:53733)
0926 12:32:07.375227 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:07.375277 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260926 12:32:08.069900 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:08.069460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:51373)
0926 12:32:08.069717 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:08.069770 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260926 12:32:08.200062 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:08.199651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:53759)
0926 12:32:08.199913 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:08.199974 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260926 12:32:08.376533 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:08.376042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:41163)
0926 12:32:08.376365 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:08.376433 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
I20260926 12:32:09.071074 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:09.070653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:53123)
0926 12:32:09.070920 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:09.070975 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260926 12:32:09.201256 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:09.200845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:36415)
0926 12:32:09.201096 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:09.201150 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260926 12:32:09.377673 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:09.377220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:47965)
0926 12:32:09.377495 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:09.377564 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260926 12:32:09.422057 24539 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:09.422186 24539 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:09.422536 24539 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
W20260926 12:32:09.423420 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
W20260926 12:32:09.423560 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
I20260926 12:32:09.423617 24143 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3d79df178a64f85b88024b1c3d71f66; no voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32
I20260926 12:32:09.423821 24539 raft_consensus.cc:2726] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:32:10.072436 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:10.071889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:60615)
0926 12:32:10.072256 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:10.072329 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260926 12:32:10.202361 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:10.201964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:37867)
0926 12:32:10.202213 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:10.202269 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260926 12:32:10.378705 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:10.378347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:45855)
0926 12:32:10.378569 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:10.378620 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260926 12:32:11.073728 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:11.073296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:45343)
0926 12:32:11.073581 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:11.073636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260926 12:32:11.203477 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:11.203103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:59105)
0926 12:32:11.203340 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:11.203392 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260926 12:32:11.379935 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:11.379479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:52797)
0926 12:32:11.379767 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:11.379827 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260926 12:32:11.628968 24546 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:11.629109 24546 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:11.629484 24546 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
W20260926 12:32:11.630360 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
W20260926 12:32:11.630518 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
I20260926 12:32:11.630584 24143 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3d79df178a64f85b88024b1c3d71f66; no voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32
I20260926 12:32:11.630770 24546 raft_consensus.cc:2726] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:32:12.074862 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:12.074462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:38663)
0926 12:32:12.074717 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:12.074768 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
I20260926 12:32:12.204619 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:12.204146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:45975)
0926 12:32:12.204456 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:12.204518 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":68,"threads_started":1}
I20260926 12:32:12.381093 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:12.380728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:58945)
0926 12:32:12.380957 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:12.381007 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260926 12:32:13.076081 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:13.075679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:56335)
0926 12:32:13.075927 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:13.075980 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260926 12:32:13.205811 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:13.205352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:49875)
0926 12:32:13.205636 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:13.205691 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260926 12:32:13.382197 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:13.381747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:56049)
0926 12:32:13.382052 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:13.382102 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":70,"threads_started":1}
I20260926 12:32:14.077178 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:14.076777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:52761)
0926 12:32:14.077030 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:14.077085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
I20260926 12:32:14.206975 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:14.206523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:59413)
0926 12:32:14.206824 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:14.206882 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260926 12:32:14.383478 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:14.382934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:33227)
0926 12:32:14.383338 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:14.383388 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":195,"threads_started":1}
I20260926 12:32:15.078495 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:15.078066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:57075)
0926 12:32:15.078338 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:15.078395 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260926 12:32:15.208276 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:15.207767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:36433)
0926 12:32:15.208037 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:15.208098 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260926 12:32:15.384599 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:15.384138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:45173)
0926 12:32:15.384449 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:15.384503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260926 12:32:16.079814 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:16.079436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:56021)
0926 12:32:16.079672 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:16.079724 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260926 12:32:16.209575 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:16.209139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:36463)
0926 12:32:16.209424 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:16.209479 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260926 12:32:16.385685 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:16.385285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:59977)
0926 12:32:16.385543 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:16.385596 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":67,"threads_started":1}
I20260926 12:32:16.622035 24562 raft_consensus.cc:482] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:16.622197 24562 raft_consensus.cc:504] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d96d825073e746b29d10361bd7d55d32" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33543 } } peers { permanent_uuid: "a3d79df178a64f85b88024b1c3d71f66" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 40015 } } peers { permanent_uuid: "73eebbb49697477b87e4fd73fa382899" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 33779 } }
I20260926 12:32:16.622654 24562 leader_election.cc:288] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543), 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779)
W20260926 12:32:16.623449 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73eebbb49697477b87e4fd73fa382899 (127.7.99.2:33779): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
W20260926 12:32:16.623589 24143 leader_election.cc:341] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d96d825073e746b29d10361bd7d55d32 (127.7.99.3:33543): Not found: Tablet not found: 63fec2d2037545409aaf1b2420d3dd06
I20260926 12:32:16.623665 24143 leader_election.cc:302] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3d79df178a64f85b88024b1c3d71f66; no voters: 73eebbb49697477b87e4fd73fa382899, d96d825073e746b29d10361bd7d55d32
I20260926 12:32:16.623831 24562 raft_consensus.cc:2726] T 63fec2d2037545409aaf1b2420d3dd06 P a3d79df178a64f85b88024b1c3d71f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:32:17.080901 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:17.080529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:60693)
0926 12:32:17.080748 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:17.080807 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260926 12:32:17.210695 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:17.210314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:53445)
0926 12:32:17.210548 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:17.210601 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260926 12:32:17.386906 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:17.386503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:34099)
0926 12:32:17.386757 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:17.386813 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260926 12:32:18.082064 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:18.081661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:49379)
0926 12:32:18.081921 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:18.081970 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
I20260926 12:32:18.211858 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:18.211476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:46671)
0926 12:32:18.211705 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:18.211757 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260926 12:32:18.387887 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:18.387514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:57201)
0926 12:32:18.387742 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:18.387795 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260926 12:32:19.083106 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:19.082718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:45471)
0926 12:32:19.082961 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:19.083013 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260926 12:32:19.213019 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:19.212622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:57825)
0926 12:32:19.212868 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:19.212921 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260926 12:32:19.389022 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:19.388608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:36649)
0926 12:32:19.388858 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:19.388909 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260926 12:32:20.084403 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:20.083847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:49125)
0926 12:32:20.084231 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:20.084295 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":138,"threads_started":1}
I20260926 12:32:20.214365 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:20.213828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:53615)
0926 12:32:20.214180 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:20.214258 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260926 12:32:20.390098 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:20.389681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.3:51545)
0926 12:32:20.389955 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:20.390008 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260926 12:32:21.085752 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:21.085267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.1:37059)
0926 12:32:21.085599 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:21.085656 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":74,"threads_started":1}
I20260926 12:32:21.215827 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:21.215398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:44559 (local address 127.7.99.2:51839)
0926 12:32:21.215662 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:21.215715 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.62:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260926 12:32:21.377513 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24086
I20260926 12:32:21.383219 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24258
I20260926 12:32:21.388429 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24393
2026-09-26T12:32:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59470 ms)
[----------] 30 tests from SecurityITest (240569 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:21 dist-test-slave-3jw2 krb5kdc[24579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:21 dist-test-slave-3jw2 krb5kdc[24579](info): set up 2 sockets
Sep 26 12:32:21 dist-test-slave-3jw2 krb5kdc[24579](info): commencing operation
krb5kdc: starting...
W20260926 12:32:22.420059 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:22 dist-test-slave-3jw2 krb5kdc[24579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:22.539497 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36823
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:22.646337 24592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:22.646615 24592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:22.646693 24592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:22.650667 24592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:22.650749 24592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:22.650799 24592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:22.650856 24592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:22.650892 24592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:22.655910 24592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:36823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:22.656898 24592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:22.657859 24592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:22.666146 24597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:22.666193 24600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:22.666147 24598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:22.666500 24592 server_base.cc:1034] running on GCE node
I20260926 12:32:22.667002 24592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:22.668042 24592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:22.669229 24592 hybrid_clock.cc:648] HybridClock initialized: now 1790425942669209 us; error 32 us; skew 500 ppm
Sep 26 12:32:22 dist-test-slave-3jw2 krb5kdc[24579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425942, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:22.671968 24592 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:22.672866 24592 webserver.cc:460] Webserver started at http://127.7.99.62:39847/ using document root <none> and password file <none>
I20260926 12:32:22.673403 24592 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:22.673488 24592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:22.673710 24592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:22.675752 24592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "0ac084645fec4a9d9fd0f36babc17d77"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "e1cb8dc6bad8cbbbb10af25b97d1d3c8"
server_key_iv: "9e8ad6b5b98e5c99b3077fdce090473b"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.676389 24592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "0ac084645fec4a9d9fd0f36babc17d77"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "e1cb8dc6bad8cbbbb10af25b97d1d3c8"
server_key_iv: "9e8ad6b5b98e5c99b3077fdce090473b"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.679970 24592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:32:22.682343 24607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:22.683441 24592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:22.683583 24592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "0ac084645fec4a9d9fd0f36babc17d77"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "e1cb8dc6bad8cbbbb10af25b97d1d3c8"
server_key_iv: "9e8ad6b5b98e5c99b3077fdce090473b"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.683691 24592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:22.702867 24592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:22.717234 24592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:22.717447 24592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:22.725029 24592 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:36823
I20260926 12:32:22.725060 24659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:36823 every 8 connection(s)
I20260926 12:32:22.725950 24592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:22.728812 24660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:22.734167 24660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: Bootstrap starting.
I20260926 12:32:22.735381 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24592
I20260926 12:32:22.735504 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:22.735821 7564 external_mini_cluster.cc:1428] Setting key cbe1a7ec90f2e1919b20d871bdfbf9e2
I20260926 12:32:22.736800 24660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:22.737689 24660 log.cc:824] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:22.739687 24660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: No bootstrap required, opened a new log
I20260926 12:32:22.743945 24660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } }
I20260926 12:32:22.744202 24660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:22.744266 24660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac084645fec4a9d9fd0f36babc17d77, State: Initialized, Role: FOLLOWER
I20260926 12:32:22.744693 24660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } }
I20260926 12:32:22.744808 24660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:22.744891 24660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 26 12:32:22 dist-test-slave-3jw2 krb5kdc[24579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:22.745008 24660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:22.746093 24660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } }
I20260926 12:32:22.746492 24660 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac084645fec4a9d9fd0f36babc17d77; no voters:
I20260926 12:32:22.746829 24660 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:22.746930 24665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:22.747144 24665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac084645fec4a9d9fd0f36babc17d77, State: Running, Role: LEADER
I20260926 12:32:22.747532 24665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } }
I20260926 12:32:22.748353 24660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:22.749272 24666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac084645fec4a9d9fd0f36babc17d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } } }
I20260926 12:32:22.749400 24666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:22.749794 24667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac084645fec4a9d9fd0f36babc17d77. Latest consensus state: current_term: 1 leader_uuid: "0ac084645fec4a9d9fd0f36babc17d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac084645fec4a9d9fd0f36babc17d77" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36823 } } }
I20260926 12:32:22.749960 24667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:22.750851 24674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:22.751307 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:22.736917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.7.99.62:36823)
0926 12:32:22.737360 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:22.737378 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:22.737418 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:32:22.738216 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:22.738235 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:22.738621 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:22.738972 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:22.739609 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:22.741129 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:22.741959 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:22.742739 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:22.742895 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:22.745508 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:22.745513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:22.745529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:22.745559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:22.747531 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:22.748013 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:22.748019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:22.748024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:22.748194 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:22.748507 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:22.748511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:22.748513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:22.749031 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:22.749875 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:22.750048 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:32:22.750353 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":146,"threads_started":1}
I20260926 12:32:22.753854 24674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:22.759230 24674 catalog_manager.cc:1347] Generated new cluster ID: aa0938e20a394681a455514e1acb0145
I20260926 12:32:22.759301 24674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:22.779763 24674 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:22.780815 24674 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:22.789717 24674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: Generated new TSK 0
I20260926 12:32:22.790490 24674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:32:22.820989 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:22.932897 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:22.933145 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:22.933187 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:22.937125 24686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:22.937176 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:22.937237 24686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:22.942287 24686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:22.943233 24686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:22.944150 24686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:22.953949 24694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:22.953975 24691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:22.954154 24692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:22.954805 24686 server_base.cc:1034] running on GCE node
I20260926 12:32:22.955317 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:22.956063 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:22.957268 24686 hybrid_clock.cc:648] HybridClock initialized: now 1790425942957201 us; error 83 us; skew 500 ppm
Sep 26 12:32:22 dist-test-slave-3jw2 krb5kdc[24579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425942, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:22.960068 24686 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:32:22.960943 24686 webserver.cc:460] Webserver started at http://127.7.99.1:42227/ using document root <none> and password file <none>
I20260926 12:32:22.961493 24686 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:22.961556 24686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:22.961838 24686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:22.963843 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "3e9c8ceb15474fe5a69c672e5f9eea02"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "2476b4d54a6749b9c1ad35475e30ba2e"
server_key_iv: "22e8b2c17ce5e019f943b0c2597d76fc"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.964442 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "3e9c8ceb15474fe5a69c672e5f9eea02"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "2476b4d54a6749b9c1ad35475e30ba2e"
server_key_iv: "22e8b2c17ce5e019f943b0c2597d76fc"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.968050 24686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:32:22.970475 24701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:22.971689 24686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:22.971825 24686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "3e9c8ceb15474fe5a69c672e5f9eea02"
format_stamp: "Formatted at 2026-09-26 12:32:22 on dist-test-slave-3jw2"
server_key: "2476b4d54a6749b9c1ad35475e30ba2e"
server_key_iv: "22e8b2c17ce5e019f943b0c2597d76fc"
server_key_version: "encryptionkey@0"
I20260926 12:32:22.971932 24686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:22.983624 24686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:22.986333 24686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:22.986526 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:22.987031 24686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:22.987915 24686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:22.987963 24686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:22.988004 24686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:22.988029 24686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:22.997541 24686 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:35111
I20260926 12:32:22.997557 24814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:35111 every 8 connection(s)
I20260926 12:32:22.998427 24686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:32:23 dist-test-slave-3jw2 krb5kdc[24579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425942, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:23.007361 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24686
I20260926 12:32:23.007499 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:23.007827 7564 external_mini_cluster.cc:1428] Setting key 0e5c9eff604d6393eb871f6d741a9004
I20260926 12:32:23.011931 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.000222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:59341 (local address 127.7.99.62:36823)
0926 12:32:23.000360 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.000364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.001163 (+ 799us) server_negotiation.cc:407] Connection header received
0926 12:32:23.001962 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.001964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.002014 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.002108 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:23.003228 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.003767 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.004937 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.005219 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.005313 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.007714 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:23.007718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:23.007720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:23.007744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:23.009573 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.010211 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.010214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.010215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.010295 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.010680 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.010685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.010689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.010924 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:23.011030 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.011608 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.011746 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:32:23.012935 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.000574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36823 (local address 127.7.99.1:59341)
0926 12:32:23.001007 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.001045 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.001754 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.002250 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.002256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.002581 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:23.003040 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.003054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.003899 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.003901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.004816 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.004824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.005348 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.005350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.005480 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.005997 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:23.006011 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:23.007540 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:23.009693 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.009702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.009717 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.010107 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.010388 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.010394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.010399 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.010581 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.011097 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:23.011105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:23.011351 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.011549 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.011868 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260926 12:32:23.014153 24815 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36823
I20260926 12:32:23.014379 24815 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:23.014830 24815 heartbeater.cc:511] Master 127.7.99.62:36823 requested a full tablet report, sending...
I20260926 12:32:23.016553 24624 ts_manager.cc:194] Registered new tserver with Master: 3e9c8ceb15474fe5a69c672e5f9eea02 (127.7.99.1:35111)
I20260926 12:32:23.018167 24624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:59341
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:32:23.039095 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:23.147884 24820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:23.148141 24820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:23.148284 24820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:23.152462 24820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:23.152515 24820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:23.152609 24820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:23.158588 24820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:23.159672 24820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:23.160796 24820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:23.170688 24825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:23.170665 24826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.170807 24820 server_base.cc:1034] running on GCE node
W20260926 12:32:23.170665 24828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.171393 24820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:23.171947 24820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:23.173110 24820 hybrid_clock.cc:648] HybridClock initialized: now 1790425943173071 us; error 48 us; skew 500 ppm
Sep 26 12:32:23 dist-test-slave-3jw2 krb5kdc[24579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425943, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:23.175997 24820 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:32:23.176990 24820 webserver.cc:460] Webserver started at http://127.7.99.2:35545/ using document root <none> and password file <none>
I20260926 12:32:23.177531 24820 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:23.177618 24820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:23.177860 24820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:23.179808 24820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "cbe302ee00794f19bd877db5e832f00e"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "ef517b3ba27d9730e3a11242e02a3496"
server_key_iv: "27d3e81decbb909068b5aeaa98615ff0"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.180466 24820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "cbe302ee00794f19bd877db5e832f00e"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "ef517b3ba27d9730e3a11242e02a3496"
server_key_iv: "27d3e81decbb909068b5aeaa98615ff0"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.184065 24820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:32:23.186487 24835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.187706 24820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:23.187882 24820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "cbe302ee00794f19bd877db5e832f00e"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "ef517b3ba27d9730e3a11242e02a3496"
server_key_iv: "27d3e81decbb909068b5aeaa98615ff0"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.187995 24820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:23.241429 24820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:23.244843 24820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:23.245040 24820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:23.245663 24820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:23.246456 24820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:23.246500 24820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.246541 24820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:23.246596 24820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.256021 24820 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:44451
I20260926 12:32:23.256038 24948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:44451 every 8 connection(s)
I20260926 12:32:23.256979 24820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:23.265797 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24820
I20260926 12:32:23.265941 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:23.266255 7564 external_mini_cluster.cc:1428] Setting key c57b51118857bd1ac98b3868ca001ebc
Sep 26 12:32:23 dist-test-slave-3jw2 krb5kdc[24579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425943, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:23.271029 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.258871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:49199 (local address 127.7.99.62:36823)
0926 12:32:23.259005 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.259008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.259841 (+ 833us) server_negotiation.cc:407] Connection header received
0926 12:32:23.260752 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.260755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.260804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.260879 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:23.262136 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.262655 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.263995 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.264277 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.264355 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.267006 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:23.267010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:23.267012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:23.267046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:23.268812 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.269290 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.269296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.269301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.269402 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.269800 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.269803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.269804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.270038 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:23.270156 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.270812 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.270912 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:32:23.272229 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.259236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36823 (local address 127.7.99.2:49199)
0926 12:32:23.259686 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.259726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.260559 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.261061 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.261068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.261496 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:23.261987 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.262000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.262807 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.262811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.263869 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.263894 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.264426 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.264429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.264584 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.265138 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:23.265156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:23.266832 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:23.268924 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.268929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.268940 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.269191 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.269510 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.269515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.269518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.269704 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.270156 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:23.270165 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:23.270510 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.270734 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.271103 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260926 12:32:23.273507 24949 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36823
I20260926 12:32:23.273780 24949 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:23.274286 24949 heartbeater.cc:511] Master 127.7.99.62:36823 requested a full tablet report, sending...
I20260926 12:32:23.275379 24624 ts_manager.cc:194] Registered new tserver with Master: cbe302ee00794f19bd877db5e832f00e (127.7.99.2:44451)
I20260926 12:32:23.276023 24624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:49199
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:32:23.295214 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:23.402568 24954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:23.402822 24954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:23.402864 24954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:23.407279 24954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:23.407333 24954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:23.407388 24954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:23.412554 24954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:23.413455 24954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:23.414350 24954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:23.423542 24962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:23.423538 24959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.423640 24954 server_base.cc:1034] running on GCE node
W20260926 12:32:23.423560 24960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.424252 24954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:23.424860 24954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:23.426069 24954 hybrid_clock.cc:648] HybridClock initialized: now 1790425943426035 us; error 43 us; skew 500 ppm
Sep 26 12:32:23 dist-test-slave-3jw2 krb5kdc[24579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425943, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:23.428874 24954 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:32:23.429678 24954 webserver.cc:460] Webserver started at http://127.7.99.3:46513/ using document root <none> and password file <none>
I20260926 12:32:23.430294 24954 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:23.430384 24954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:23.430627 24954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:23.432653 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "d8ca9ec431c8475c85380232a9aafde0"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "6faae8209ff7a155497e934a6c6676f1"
server_key_iv: "686edc2eb9612f8c52f65bbe4a50ca84"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.433214 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "d8ca9ec431c8475c85380232a9aafde0"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "6faae8209ff7a155497e934a6c6676f1"
server_key_iv: "686edc2eb9612f8c52f65bbe4a50ca84"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.436817 24954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:32:23.439155 24969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.440323 24954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:23.440452 24954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "d8ca9ec431c8475c85380232a9aafde0"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "6faae8209ff7a155497e934a6c6676f1"
server_key_iv: "686edc2eb9612f8c52f65bbe4a50ca84"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.440534 24954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:23.467855 24954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:23.470881 24954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:23.471083 24954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:23.471666 24954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:23.472556 24954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:23.472608 24954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.472680 24954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:23.472718 24954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.481789 24954 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:42501
I20260926 12:32:23.481848 25082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:42501 every 8 connection(s)
I20260926 12:32:23.482671 24954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:23.491328 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 24954
I20260926 12:32:23.491487 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:23.491973 7564 external_mini_cluster.cc:1428] Setting key 4580c20ab5dd8b7f6354b960464c5cdb
Sep 26 12:32:23 dist-test-slave-3jw2 krb5kdc[24579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425943, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:23.496807 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.484469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:54467 (local address 127.7.99.62:36823)
0926 12:32:23.484635 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.484640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.485423 (+ 783us) server_negotiation.cc:407] Connection header received
0926 12:32:23.486683 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.486686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.486732 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.486805 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:23.488114 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.488710 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.489833 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.490048 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.490121 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.493215 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:23.493219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:23.493221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:23.493252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:23.494777 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.495289 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.495292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.495294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.495374 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.495706 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.495710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.495712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.495888 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:23.495996 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.496528 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.496640 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:32:23.497650 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.484862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36823 (local address 127.7.99.3:54467)
0926 12:32:23.485282 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.485321 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.486428 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.486960 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.486966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.487447 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:23.487986 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.488000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.488819 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.488822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.489728 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.489737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.490203 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.490206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.490338 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.491000 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:23.491039 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:23.493057 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:23.494907 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.494913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.494927 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.495179 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.495466 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:23.495471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:23.495473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:23.495613 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:23.495992 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:23.495998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:23.496296 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.496476 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.496788 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260926 12:32:23.498803 25083 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36823
I20260926 12:32:23.499039 25083 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:23.499637 25083 heartbeater.cc:511] Master 127.7.99.62:36823 requested a full tablet report, sending...
I20260926 12:32:23.500715 24624 ts_manager.cc:194] Registered new tserver with Master: d8ca9ec431c8475c85380232a9aafde0 (127.7.99.3:42501)
I20260926 12:32:23.501303 24624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:54467
I20260926 12:32:23.506105 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:23.515257 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.508010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.7.99.62:36823)
0926 12:32:23.508140 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.508144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.508423 (+ 279us) server_negotiation.cc:407] Connection header received
0926 12:32:23.508541 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.508544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.508589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.508675 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:23.509350 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.509883 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.510740 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.510960 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.511067 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.512471 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:23.512475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:23.512478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:23.512507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:23.513627 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.514033 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.514036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.514038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.514121 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:23.514391 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:23.514395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:23.514396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:23.514623 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:23.514766 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.514985 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.515099 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:23.517916 24624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:23.521080 24624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:23.535264 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.529584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44451 (local address 127.0.0.1:33688)
0926 12:32:23.530213 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.530259 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.530408 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.531058 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.531064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.531102 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:23.531246 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.531253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.532548 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.532551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.533761 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.533770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.534711 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.534715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.534831 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.534871 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.534992 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.535103 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260926 12:32:23.535302 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.529720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.7.99.2:44451)
0926 12:32:23.530059 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.530071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.530547 (+ 476us) server_negotiation.cc:407] Connection header received
0926 12:32:23.530632 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.530638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.530876 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.531042 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:23.531364 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.532422 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.533878 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.534595 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.534700 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.534849 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.535035 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.535140 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":135,"threads_started":1}
I20260926 12:32:23.535584 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.529716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:42501 (local address 127.0.0.1:38508)
0926 12:32:23.530917 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.530932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.531007 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.531538 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.531541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.531558 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:23.531647 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.531653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.532949 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.532953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.534149 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.534158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.535212 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.535218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.535329 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.535355 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.535411 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.535478 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"thread_start_us":103,"threads_started":1}
I20260926 12:32:23.536123 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.530334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35111 (local address 127.0.0.1:48440)
0926 12:32:23.531192 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.531205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.531299 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.531866 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.531869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.531884 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:23.531956 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.531961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.533367 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.533370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.534433 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.534440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.535665 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.535670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.535761 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.535782 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.535932 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.536010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":96,"threads_started":1}
I20260926 12:32:23.536149 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.530733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.7.99.3:42501)
0926 12:32:23.531123 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.531134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.531155 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:32:23.531225 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.531229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.531399 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.531550 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:23.531738 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.532804 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.534279 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.535080 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.535662 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.535844 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.535896 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.535978 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
I20260926 12:32:23.536751 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.530497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.7.99.1:35111)
0926 12:32:23.530876 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.530883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.531285 (+ 402us) server_negotiation.cc:407] Connection header received
0926 12:32:23.531443 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.531448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.531618 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.531742 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:23.532083 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.533238 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.534543 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.535358 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.535472 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.535591 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.536502 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.536600 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":88,"threads_started":1}
I20260926 12:32:23.538679 24883 tablet_service.cc:1467] Processing CreateTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 (DEFAULT_TABLE table=test-table [id=ac22f6bef75646ab87b51f78b65578cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:23.538786 25017 tablet_service.cc:1467] Processing CreateTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 (DEFAULT_TABLE table=test-table [id=ac22f6bef75646ab87b51f78b65578cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:23.539096 24749 tablet_service.cc:1467] Processing CreateTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 (DEFAULT_TABLE table=test-table [id=ac22f6bef75646ab87b51f78b65578cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:23.539767 24883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:23.539870 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:23.539938 24749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:23.546092 25100 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Bootstrap starting.
I20260926 12:32:23.547495 25102 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Bootstrap starting.
I20260926 12:32:23.547541 25101 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: Bootstrap starting.
I20260926 12:32:23.548573 25100 tablet_bootstrap.cc:654] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:23.549474 25101 tablet_bootstrap.cc:654] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:23.549678 25102 tablet_bootstrap.cc:654] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:23.549973 25100 log.cc:824] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:23.550307 25101 log.cc:824] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:23.551043 25102 log.cc:824] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:23.552186 25100 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: No bootstrap required, opened a new log
I20260926 12:32:23.552397 25100 ts_tablet_manager.cc:1397] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:32:23.552946 25102 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: No bootstrap required, opened a new log
I20260926 12:32:23.553136 25102 ts_tablet_manager.cc:1397] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 12:32:23.556273 25100 raft_consensus.cc:348] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.556568 25100 raft_consensus.cc:374] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:23.556664 25100 raft_consensus.cc:729] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe302ee00794f19bd877db5e832f00e, State: Initialized, Role: FOLLOWER
I20260926 12:32:23.556763 25102 raft_consensus.cc:348] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.557000 25102 raft_consensus.cc:374] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:23.557052 25102 raft_consensus.cc:729] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ca9ec431c8475c85380232a9aafde0, State: Initialized, Role: FOLLOWER
I20260926 12:32:23.557128 25100 consensus_queue.cc:260] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.557596 25102 consensus_queue.cc:260] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.557962 25101 tablet_bootstrap.cc:492] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: No bootstrap required, opened a new log
I20260926 12:32:23.558137 25101 ts_tablet_manager.cc:1397] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 12:32:23.558135 25100 ts_tablet_manager.cc:1428] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:32:23.558501 24949 heartbeater.cc:503] Master 127.7.99.62:36823 was elected leader, sending a full tablet report...
I20260926 12:32:23.558511 25102 ts_tablet_manager.cc:1428] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:32:23.558845 25083 heartbeater.cc:503] Master 127.7.99.62:36823 was elected leader, sending a full tablet report...
I20260926 12:32:23.562453 25101 raft_consensus.cc:348] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.562724 25101 raft_consensus.cc:374] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:23.562789 25101 raft_consensus.cc:729] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9c8ceb15474fe5a69c672e5f9eea02, State: Initialized, Role: FOLLOWER
I20260926 12:32:23.563079 25101 consensus_queue.cc:260] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.563697 24815 heartbeater.cc:503] Master 127.7.99.62:36823 was elected leader, sending a full tablet report...
I20260926 12:32:23.563773 25101 ts_tablet_manager.cc:1428] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:32:23.619838 25107 raft_consensus.cc:482] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:23.620067 25107 raft_consensus.cc:504] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.621285 25107 leader_election.cc:288] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe302ee00794f19bd877db5e832f00e (127.7.99.2:44451), 3e9c8ceb15474fe5a69c672e5f9eea02 (127.7.99.1:35111)
I20260926 12:32:23.625146 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.621639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:59269 (local address 127.7.99.2:44451)
0926 12:32:23.621753 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.621758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.621864 (+ 106us) server_negotiation.cc:407] Connection header received
0926 12:32:23.621948 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.621950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.621993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.622090 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:23.622357 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.622860 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.623880 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.624526 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.624615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.624694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.624983 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.625047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:23.625166 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.621515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:44451 (local address 127.7.99.3:59269)
0926 12:32:23.621736 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.621753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.621861 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.622104 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.622107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.622126 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:23.622243 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.622248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.622966 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.622972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.623723 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.623732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.624749 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.624753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.624857 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.624878 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.624955 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.625030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 12:32:23.625736 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.621686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:59957 (local address 127.7.99.1:35111)
0926 12:32:23.621810 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.621815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.622067 (+ 252us) server_negotiation.cc:407] Connection header received
0926 12:32:23.622211 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.622214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.622255 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.622335 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:23.622583 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.623066 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.624472 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.625082 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.625162 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.625233 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.625576 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.625651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:32:23.625734 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.621576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35111 (local address 127.7.99.3:59957)
0926 12:32:23.621964 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.621984 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:23.622092 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:23.622343 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:23.622345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:23.622359 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:23.622480 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.622485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.623216 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.623219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.624334 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:23.624345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.625341 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.625347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:23.625441 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.625466 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:32:23.625555 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:32:23.625613 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":132,"threads_started":1}
I20260926 12:32:23.625828 24903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cbd5e9b6f24dbd9638a8f0bc1f0a16" candidate_uuid: "d8ca9ec431c8475c85380232a9aafde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe302ee00794f19bd877db5e832f00e" is_pre_election: true
I20260926 12:32:23.626163 24903 raft_consensus.cc:2454] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ca9ec431c8475c85380232a9aafde0 in term 0.
I20260926 12:32:23.626412 24769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cbd5e9b6f24dbd9638a8f0bc1f0a16" candidate_uuid: "d8ca9ec431c8475c85380232a9aafde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" is_pre_election: true
I20260926 12:32:23.626663 24769 raft_consensus.cc:2454] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ca9ec431c8475c85380232a9aafde0 in term 0.
I20260926 12:32:23.626648 24971 leader_election.cc:302] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbe302ee00794f19bd877db5e832f00e, d8ca9ec431c8475c85380232a9aafde0; no voters:
I20260926 12:32:23.626936 25107 raft_consensus.cc:2781] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:23.627032 25107 raft_consensus.cc:482] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:23.627087 25107 raft_consensus.cc:3037] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:23.628094 25107 raft_consensus.cc:504] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.628481 25107 leader_election.cc:288] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [CANDIDATE]: Term 1 election: Requested vote from peers cbe302ee00794f19bd877db5e832f00e (127.7.99.2:44451), 3e9c8ceb15474fe5a69c672e5f9eea02 (127.7.99.1:35111)
I20260926 12:32:23.628825 24769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cbd5e9b6f24dbd9638a8f0bc1f0a16" candidate_uuid: "d8ca9ec431c8475c85380232a9aafde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02"
I20260926 12:32:23.628822 24903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cbd5e9b6f24dbd9638a8f0bc1f0a16" candidate_uuid: "d8ca9ec431c8475c85380232a9aafde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe302ee00794f19bd877db5e832f00e"
I20260926 12:32:23.628948 24769 raft_consensus.cc:3037] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:23.628958 24903 raft_consensus.cc:3037] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:23.630017 24769 raft_consensus.cc:2454] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ca9ec431c8475c85380232a9aafde0 in term 1.
I20260926 12:32:23.630069 24903 raft_consensus.cc:2454] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ca9ec431c8475c85380232a9aafde0 in term 1.
I20260926 12:32:23.630323 24973 leader_election.cc:302] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e9c8ceb15474fe5a69c672e5f9eea02, d8ca9ec431c8475c85380232a9aafde0; no voters:
I20260926 12:32:23.630483 25107 raft_consensus.cc:2781] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:23.630695 25107 raft_consensus.cc:686] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 1 LEADER]: Becoming Leader. State: Replica: d8ca9ec431c8475c85380232a9aafde0, State: Running, Role: LEADER
I20260926 12:32:23.631016 25107 consensus_queue.cc:237] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } }
I20260926 12:32:23.634215 24624 catalog_manager.cc:5495] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ca9ec431c8475c85380232a9aafde0 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "d8ca9ec431c8475c85380232a9aafde0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ca9ec431c8475c85380232a9aafde0" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 } health_report { overall_health: UNKNOWN } } }
I20260926 12:32:23.656766 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.653169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.7.99.3:42501)
0926 12:32:23.653300 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.653304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.653406 (+ 102us) server_negotiation.cc:407] Connection header received
0926 12:32:23.653569 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.653571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.653626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.653705 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:23.653943 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.654429 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.655370 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.655563 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.655634 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.655930 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:23.656416 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:23.656504 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.656611 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.656659 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:32:23.662406 24769 raft_consensus.cc:1261] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 1 FOLLOWER]: Refusing update from remote peer d8ca9ec431c8475c85380232a9aafde0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:23.662432 24903 raft_consensus.cc:1261] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 1 FOLLOWER]: Refusing update from remote peer d8ca9ec431c8475c85380232a9aafde0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:23.663085 25110 consensus_queue.cc:1035] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e9c8ceb15474fe5a69c672e5f9eea02" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:23.663276 25107 consensus_queue.cc:1035] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe302ee00794f19bd877db5e832f00e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:23.673677 25114 mvcc.cc:205] Tried to move back new op lower bound from 7333584665234284544 to 7333584665115111424. Current Snapshot: MvccSnapshot[applied={T|T < 7333584665234284544}]
I20260926 12:32:23.678808 24623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:23.678959 24623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:23.681299 24623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac084645fec4a9d9fd0f36babc17d77: Sending DeleteTablet for 3 replicas of tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16
I20260926 12:32:23.682034 24883 tablet_service.cc:1514] Processing DeleteTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:33688
I20260926 12:32:23.682170 25017 tablet_service.cc:1514] Processing DeleteTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260926 12:32:23.682245 24749 tablet_service.cc:1514] Processing DeleteTablet for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:23 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:48440
I20260926 12:32:23.682691 25122 tablet_replica.cc:330] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: stopping tablet replica
I20260926 12:32:23.683027 25122 raft_consensus.cc:2229] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:23.683077 25124 tablet_replica.cc:330] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02: stopping tablet replica
I20260926 12:32:23.683332 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24686
I20260926 12:32:23.683384 25122 raft_consensus.cc:2258] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:23.683421 25124 raft_consensus.cc:2229] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:23.683694 25124 raft_consensus.cc:2258] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P 3e9c8ceb15474fe5a69c672e5f9eea02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:23.684026 25123 tablet_replica.cc:330] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: stopping tablet replica
I20260926 12:32:23.684340 25123 raft_consensus.cc:2229] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:23.684671 25123 raft_consensus.cc:2258] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:23.685205 25122 ts_tablet_manager.cc:1905] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:23.686120 25123 ts_tablet_manager.cc:1905] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:23.687968 25122 ts_tablet_manager.cc:1918] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:23.688138 25122 log.cc:1196] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/d2cbd5e9b6f24dbd9638a8f0bc1f0a16
I20260926 12:32:23.688504 25122 ts_tablet_manager.cc:1939] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P cbe302ee00794f19bd877db5e832f00e: Deleting consensus metadata
I20260926 12:32:23.689452 25123 ts_tablet_manager.cc:1918] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:23.689548 25123 log.cc:1196] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/d2cbd5e9b6f24dbd9638a8f0bc1f0a16
I20260926 12:32:23.689759 24609 catalog_manager.cc:4841] TS cbe302ee00794f19bd877db5e832f00e (127.7.99.2:44451): tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 (table test-table [id=ac22f6bef75646ab87b51f78b65578cf]) successfully deleted
I20260926 12:32:23.690028 25123 ts_tablet_manager.cc:1939] T d2cbd5e9b6f24dbd9638a8f0bc1f0a16 P d8ca9ec431c8475c85380232a9aafde0: Deleting consensus metadata
I20260926 12:32:23.691118 24611 catalog_manager.cc:4841] TS d8ca9ec431c8475c85380232a9aafde0 (127.7.99.3:42501): tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16 (table test-table [id=ac22f6bef75646ab87b51f78b65578cf]) successfully deleted
W20260926 12:32:23.693907 24611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:35111 (error 108)
I20260926 12:32:23.694063 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24820
I20260926 12:32:23.695843 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.695490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35111 (local address 127.0.0.1:48448)
0926 12:32:23.695660 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:23.695740 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 12:32:23.696015 24611 catalog_manager.cc:4584] TS 3e9c8ceb15474fe5a69c672e5f9eea02 (127.7.99.1:35111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2cbd5e9b6f24dbd9638a8f0bc1f0a16: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35111: connect: Connection refused (error 111)
I20260926 12:32:23.701608 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24954
I20260926 12:32:23.707343 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 24592
2026-09-26T12:32:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T12:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:23Z Disabled control of system clock
I20260926 12:32:23.737051 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36829
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:23.852669 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:23.852945 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:23.852985 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:23.856976 25129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:23.857038 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:23.857056 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:23.857076 25129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:23.857095 25129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:23.861828 25129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:36829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:23.862941 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:23.864431 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:23.872995 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.873152 25129 server_base.cc:1034] running on GCE node
W20260926 12:32:23.873031 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:23.873031 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:23.873912 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:23.874989 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:23.876214 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790425943876188 us; error 33 us; skew 500 ppm
I20260926 12:32:23.878057 25129 webserver.cc:460] Webserver started at http://127.7.99.62:38557/ using document root <none> and password file <none>
I20260926 12:32:23.878638 25129 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:23.878728 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:23.878959 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:23.881049 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "01c2ff6ca3694f54a63c6c6e5a739f02"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "9592fcae7993dbe3a1dcd94a903bb0ee"
server_key_iv: "ab9afe3d7a225320f64dfa3444cba062"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.881621 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "01c2ff6ca3694f54a63c6c6e5a739f02"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "9592fcae7993dbe3a1dcd94a903bb0ee"
server_key_iv: "ab9afe3d7a225320f64dfa3444cba062"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.885290 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:32:23.887646 25143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:23.888960 25129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:23.889101 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "01c2ff6ca3694f54a63c6c6e5a739f02"
format_stamp: "Formatted at 2026-09-26 12:32:23 on dist-test-slave-3jw2"
server_key: "9592fcae7993dbe3a1dcd94a903bb0ee"
server_key_iv: "ab9afe3d7a225320f64dfa3444cba062"
server_key_version: "encryptionkey@0"
I20260926 12:32:23.889219 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:23.905368 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:23.906090 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:23.906299 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:23.913255 25129 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:36829
I20260926 12:32:23.913269 25195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:36829 every 8 connection(s)
I20260926 12:32:23.914187 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:23.917284 25196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:23.922698 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25129
I20260926 12:32:23.922708 25196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: Bootstrap starting.
I20260926 12:32:23.922869 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:23.923156 7564 external_mini_cluster.cc:1428] Setting key bfb8d68453b9f1c98bf6f360ba119ac4
I20260926 12:32:23.925344 25196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:23.926075 25196 log.cc:824] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:23.927850 25196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: No bootstrap required, opened a new log
I20260926 12:32:23.931397 25196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } }
I20260926 12:32:23.931660 25196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:23.931728 25196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c2ff6ca3694f54a63c6c6e5a739f02, State: Initialized, Role: FOLLOWER
I20260926 12:32:23.932289 25196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } }
I20260926 12:32:23.932418 25196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:23.932502 25196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:23.932610 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:23.924535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.7.99.62:36829)
0926 12:32:23.925185 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:23.925202 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:23.925237 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:32:23.925909 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:23.925926 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:23.926371 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:23.926701 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:23.927127 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.928178 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.929170 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:23.929716 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:23.929909 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:23.930750 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:23.930756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:23.930782 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:23.931146 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:23.931200 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:23.931220 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:23.931335 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:23.931453 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 12:32:23.931729 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260926 12:32:23.932615 25196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:23.933610 25196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } }
I20260926 12:32:23.933950 25196 leader_election.cc:302] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c2ff6ca3694f54a63c6c6e5a739f02; no voters:
I20260926 12:32:23.934264 25196 leader_election.cc:288] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:23.934383 25201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:23.934623 25201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [term 1 LEADER]: Becoming Leader. State: Replica: 01c2ff6ca3694f54a63c6c6e5a739f02, State: Running, Role: LEADER
I20260926 12:32:23.934947 25201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } }
I20260926 12:32:23.935427 25196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:23.936035 25201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c2ff6ca3694f54a63c6c6e5a739f02. Latest consensus state: current_term: 1 leader_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } } }
I20260926 12:32:23.936141 25201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:23.936527 25207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:23.936637 25203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2ff6ca3694f54a63c6c6e5a739f02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 36829 } } }
I20260926 12:32:23.936806 25203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:23.939450 25207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:23.944521 25207 catalog_manager.cc:1347] Generated new cluster ID: 10f0a34ba057492cac1ded090e8f961c
I20260926 12:32:23.944587 25207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:23.966946 25207 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:23.968255 25207 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:23.982710 25207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: Generated new TSK 0
I20260926 12:32:23.983526 25207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:23.994779 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--builtin_ntp_servers=127.7.99.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:24.104364 25220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:24.104594 25220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:24.104642 25220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:24.108556 25220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:24.108599 25220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:24.108650 25220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:24.113247 25220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:24.114181 25220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:24.115458 25220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:24.124745 25226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:24.124754 25228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:24.124763 25225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:24.124953 25220 server_base.cc:1034] running on GCE node
I20260926 12:32:24.125451 25220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:24.126061 25220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:24.127259 25220 hybrid_clock.cc:648] HybridClock initialized: now 1790425944127240 us; error 29 us; skew 500 ppm
I20260926 12:32:24.129074 25220 webserver.cc:460] Webserver started at http://127.7.99.1:35003/ using document root <none> and password file <none>
I20260926 12:32:24.129705 25220 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:24.129799 25220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:24.130045 25220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:24.131997 25220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "23accffc4b6d4f36938fc0b2b9785588"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "e7886a521532cd2729b5f54c69e9db2b"
server_key_iv: "03d9809ad64ac3914f6b5bd4f2cfdba8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.132601 25220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "23accffc4b6d4f36938fc0b2b9785588"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "e7886a521532cd2729b5f54c69e9db2b"
server_key_iv: "03d9809ad64ac3914f6b5bd4f2cfdba8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.136346 25220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:32:24.138671 25234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.139830 25220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:24.139966 25220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "23accffc4b6d4f36938fc0b2b9785588"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "e7886a521532cd2729b5f54c69e9db2b"
server_key_iv: "03d9809ad64ac3914f6b5bd4f2cfdba8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.140074 25220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:24.164817 25220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:24.165529 25220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:24.165730 25220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:24.166334 25220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:24.167292 25220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:24.167363 25220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.167433 25220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:24.167480 25220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.176317 25220 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:38727
I20260926 12:32:24.176373 25347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:38727 every 8 connection(s)
I20260926 12:32:24.177276 25220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:24.180984 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25220
I20260926 12:32:24.181087 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:24.181421 7564 external_mini_cluster.cc:1428] Setting key cda240783f18e70d039fdf6643c3f101
I20260926 12:32:24.183661 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--builtin_ntp_servers=127.7.99.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:24.188251 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.179252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:36785 (local address 127.7.99.62:36829)
0926 12:32:24.179422 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.179427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.180516 (+ 1089us) server_negotiation.cc:407] Connection header received
0926 12:32:24.182226 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.182230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.182283 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.182361 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:24.183376 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.184244 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.186007 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.186217 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.186299 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.187692 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:24.187698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:24.187703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:24.187722 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:24.187744 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:24.187753 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:24.187817 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.188036 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.188084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":70}
I20260926 12:32:24.189610 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.179561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36829 (local address 127.7.99.1:36785)
0926 12:32:24.180343 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.180399 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.181975 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.182519 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.182526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.182625 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:24.183226 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.183240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.184417 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.184420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.185860 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.185874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.186378 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.186386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.186646 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.187326 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:24.187330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:24.187392 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:24.187482 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:24.187855 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:24.187869 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:24.187879 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.188037 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.188395 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260926 12:32:24.190868 25348 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36829
I20260926 12:32:24.191145 25348 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:24.191823 25348 heartbeater.cc:511] Master 127.7.99.62:36829 requested a full tablet report, sending...
I20260926 12:32:24.193516 25160 ts_manager.cc:194] Registered new tserver with Master: 23accffc4b6d4f36938fc0b2b9785588 (127.7.99.1:38727)
I20260926 12:32:24.194665 25160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.1:36785
W20260926 12:32:24.295576 25351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:24.295877 25351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:24.295948 25351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:24.299940 25351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:24.300025 25351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:24.300120 25351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:24.304944 25351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:24.306012 25351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:24.307390 25351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:24.317060 25357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:24.317060 25359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:24.317111 25356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:24.317137 25351 server_base.cc:1034] running on GCE node
I20260926 12:32:24.317687 25351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:24.318313 25351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:24.319509 25351 hybrid_clock.cc:648] HybridClock initialized: now 1790425944319499 us; error 31 us; skew 500 ppm
I20260926 12:32:24.321484 25351 webserver.cc:460] Webserver started at http://127.7.99.2:39173/ using document root <none> and password file <none>
I20260926 12:32:24.322057 25351 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:24.322146 25351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:24.322417 25351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:24.324529 25351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "667d6dfe1ad74c9bb0533ca7bf32327b"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "bb106befaba6e8a8adb5dc9617ec3cb0"
server_key_iv: "9ac317eb61820e9dd61e600a3d783aa8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.325088 25351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "667d6dfe1ad74c9bb0533ca7bf32327b"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "bb106befaba6e8a8adb5dc9617ec3cb0"
server_key_iv: "9ac317eb61820e9dd61e600a3d783aa8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.328604 25351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:32:24.331063 25365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.332346 25351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:24.332473 25351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "667d6dfe1ad74c9bb0533ca7bf32327b"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "bb106befaba6e8a8adb5dc9617ec3cb0"
server_key_iv: "9ac317eb61820e9dd61e600a3d783aa8"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.332578 25351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:24.354631 25351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:24.355378 25351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:24.355583 25351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:24.356213 25351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:24.357173 25351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:24.357247 25351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.357321 25351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:24.357365 25351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.366982 25351 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:41823
I20260926 12:32:24.367003 25478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:41823 every 8 connection(s)
I20260926 12:32:24.367921 25351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:24.370229 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25351
I20260926 12:32:24.370378 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:24.370677 7564 external_mini_cluster.cc:1428] Setting key 913a41c5818cc282879ff6bc3dc6169a
I20260926 12:32:24.372776 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--builtin_ntp_servers=127.7.99.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:24.377696 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.369785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:55711 (local address 127.7.99.62:36829)
0926 12:32:24.369945 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.369950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.371282 (+ 1332us) server_negotiation.cc:407] Connection header received
0926 12:32:24.372913 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.372916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.372970 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.373048 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:24.373946 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.374480 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.375636 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.375868 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.375982 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.377199 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:24.377202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:24.377205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:24.377220 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:24.377237 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:24.377246 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:24.377328 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.377491 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.377551 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:32:24.378546 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.370381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36829 (local address 127.7.99.2:55711)
0926 12:32:24.371121 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.371173 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.372702 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.373217 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.373224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.373311 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:24.373799 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.373812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.374625 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.374628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.375510 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.375519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.376020 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.376024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.376258 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.376925 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:24.376927 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:24.376966 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:24.377027 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:24.377332 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:24.377344 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:24.377350 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.377480 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.377780 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260926 12:32:24.379691 25479 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36829
I20260926 12:32:24.379954 25479 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:24.380667 25479 heartbeater.cc:511] Master 127.7.99.62:36829 requested a full tablet report, sending...
I20260926 12:32:24.381850 25160 ts_manager.cc:194] Registered new tserver with Master: 667d6dfe1ad74c9bb0533ca7bf32327b (127.7.99.2:41823)
I20260926 12:32:24.382730 25160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.2:55711
W20260926 12:32:24.485585 25482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:24.485839 25482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:24.485879 25482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:24.489914 25482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:24.489964 25482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:24.490018 25482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:24.494681 25482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:24.495654 25482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:24.497045 25482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:24.507344 25487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:24.507493 25482 server_base.cc:1034] running on GCE node
W20260926 12:32:24.507344 25488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:24.507347 25490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:24.508119 25482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:24.508809 25482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:24.509964 25482 hybrid_clock.cc:648] HybridClock initialized: now 1790425944509932 us; error 44 us; skew 500 ppm
I20260926 12:32:24.511713 25482 webserver.cc:460] Webserver started at http://127.7.99.3:42181/ using document root <none> and password file <none>
I20260926 12:32:24.512334 25482 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:24.512400 25482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:24.512624 25482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:24.514605 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "e60ca9005a4f42b29d5274bb2fb93b03"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "c44b6618bb128f87e8db2e26d59ca06c"
server_key_iv: "98ce821d0a1b794ef3214d4bfc1d0827"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.515153 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "e60ca9005a4f42b29d5274bb2fb93b03"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "c44b6618bb128f87e8db2e26d59ca06c"
server_key_iv: "98ce821d0a1b794ef3214d4bfc1d0827"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.518730 25482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:32:24.521085 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.522346 25482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:24.522473 25482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "e60ca9005a4f42b29d5274bb2fb93b03"
format_stamp: "Formatted at 2026-09-26 12:32:24 on dist-test-slave-3jw2"
server_key: "c44b6618bb128f87e8db2e26d59ca06c"
server_key_iv: "98ce821d0a1b794ef3214d4bfc1d0827"
server_key_version: "encryptionkey@0"
I20260926 12:32:24.522583 25482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:24.547448 25482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:24.548151 25482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:24.548395 25482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:24.549015 25482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:24.549944 25482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:24.550012 25482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.550078 25482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:24.550124 25482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:24.558643 25482 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33563
I20260926 12:32:24.558692 25609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33563 every 8 connection(s)
I20260926 12:32:24.559510 25482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:24.568121 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.561241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:49225 (local address 127.7.99.62:36829)
0926 12:32:24.561407 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.561411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.562439 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 12:32:24.563753 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.563756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.563800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.563874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:24.564738 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.565260 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.566353 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.566552 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.566625 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.567720 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:24.567723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:24.567725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:24.567737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:24.567754 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:24.567762 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:24.567819 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.567961 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.568009 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:32:24.569043 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25482
I20260926 12:32:24.569110 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.561692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:36829 (local address 127.7.99.3:49225)
0926 12:32:24.562304 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.562342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.563575 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.564035 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.564041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.564122 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:24.564618 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.564631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.565386 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.565388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.566243 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.566251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.566676 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.566679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.566813 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.567433 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:24.567435 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:24.567474 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:24.567533 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:24.567839 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:24.567845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:24.567849 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.567944 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.568149 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260926 12:32:24.569160 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:24.569470 7564 external_mini_cluster.cc:1428] Setting key ee614c329138a5adc2f1040cffb68a46
I20260926 12:32:24.570259 25610 heartbeater.cc:347] Connected to a master server at 127.7.99.62:36829
I20260926 12:32:24.570515 25610 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:24.571038 25610 heartbeater.cc:511] Master 127.7.99.62:36829 requested a full tablet report, sending...
I20260926 12:32:24.572062 25160 ts_manager.cc:194] Registered new tserver with Master: e60ca9005a4f42b29d5274bb2fb93b03 (127.7.99.3:33563)
I20260926 12:32:24.572628 25160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.3:49225
I20260926 12:32:24.583205 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:24.589056 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.585124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.7.99.62:36829)
0926 12:32:24.585255 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.585258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.585490 (+ 232us) server_negotiation.cc:407] Connection header received
0926 12:32:24.585648 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.585651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.585703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.585783 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:24.586052 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.586584 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.587426 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.587659 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.587759 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.588598 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:24.588602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:24.588604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:24.588617 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:24.588637 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:24.588653 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:24.588726 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.588848 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.588902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:32:24.591775 25160 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:24.594918 25160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:24.609215 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.603076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.7.99.1:38727)
0926 12:32:24.603571 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.603585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.604073 (+ 488us) server_negotiation.cc:407] Connection header received
0926 12:32:24.604339 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.604345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.604554 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.604683 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:24.605206 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.606338 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.607635 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.608491 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.608599 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.608793 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.608942 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.609050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":145,"threads_started":1}
I20260926 12:32:24.609207 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.602931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38727 (local address 127.0.0.1:42728)
0926 12:32:24.603979 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.603993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.604145 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.604864 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.604869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.604887 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:24.605086 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.605092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.606421 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.606424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.607511 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.607520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.608624 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.608631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.608748 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.608785 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.608901 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.609020 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":50,"thread_start_us":98,"threads_started":1}
I20260926 12:32:24.610100 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.603723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.7.99.3:33563)
0926 12:32:24.604302 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.604315 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.604336 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:32:24.604398 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.604402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.604554 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.604693 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:24.605036 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.606306 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.607962 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.608578 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.608695 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.608877 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.609839 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.609977 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":129,"threads_started":1}
I20260926 12:32:24.610316 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.603312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33563 (local address 127.0.0.1:34658)
0926 12:32:24.603796 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.603845 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.603964 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.604723 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.604729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.604751 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:24.604899 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.604907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.606451 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.606456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.607848 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.607857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.608936 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.608939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.609017 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.609037 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.609104 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.609158 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":145,"thread_start_us":100,"threads_started":1}
I20260926 12:32:24.611070 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.603650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:41823 (local address 127.0.0.1:40954)
0926 12:32:24.604212 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.604229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.604316 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.604883 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.604885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.604897 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:24.605085 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.605092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.607772 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.607776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.608885 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.608899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.610697 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.610700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.610764 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.610783 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.610874 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.610941 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":50,"thread_start_us":95,"threads_started":1}
I20260926 12:32:24.611153 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.604055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.7.99.2:41823)
0926 12:32:24.604477 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.604486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.604503 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:32:24.604561 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.604565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.604763 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.604905 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:24.606479 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.607655 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.609025 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.610339 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.610479 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.610644 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.610905 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.610994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260926 12:32:24.612293 25282 tablet_service.cc:1467] Processing CreateTablet for tablet 979a5267ffd84f7a9947975a90524934 (DEFAULT_TABLE table=test-table [id=d95bfc722a104abcbe28f9c291f42a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:24.613497 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a5267ffd84f7a9947975a90524934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:24.614974 25544 tablet_service.cc:1467] Processing CreateTablet for tablet 979a5267ffd84f7a9947975a90524934 (DEFAULT_TABLE table=test-table [id=d95bfc722a104abcbe28f9c291f42a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:24.615357 25413 tablet_service.cc:1467] Processing CreateTablet for tablet 979a5267ffd84f7a9947975a90524934 (DEFAULT_TABLE table=test-table [id=d95bfc722a104abcbe28f9c291f42a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:24.615968 25544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a5267ffd84f7a9947975a90524934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:24.616281 25413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a5267ffd84f7a9947975a90524934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:24.619783 25627 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Bootstrap starting.
I20260926 12:32:24.621626 25627 tablet_bootstrap.cc:654] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:24.622332 25627 log.cc:824] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:24.622892 25628 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Bootstrap starting.
I20260926 12:32:24.623932 25630 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Bootstrap starting.
I20260926 12:32:24.624466 25627 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: No bootstrap required, opened a new log
I20260926 12:32:24.624646 25627 ts_tablet_manager.cc:1397] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:32:24.625087 25628 tablet_bootstrap.cc:654] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:24.625877 25630 tablet_bootstrap.cc:654] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:24.625873 25628 log.cc:824] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:24.626601 25630 log.cc:824] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:24.627866 25628 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: No bootstrap required, opened a new log
I20260926 12:32:24.628043 25628 ts_tablet_manager.cc:1397] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:32:24.628783 25630 tablet_bootstrap.cc:492] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: No bootstrap required, opened a new log
I20260926 12:32:24.628952 25630 ts_tablet_manager.cc:1397] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:32:24.629029 25627 raft_consensus.cc:348] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.629273 25627 raft_consensus.cc:374] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:24.629333 25627 raft_consensus.cc:729] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23accffc4b6d4f36938fc0b2b9785588, State: Initialized, Role: FOLLOWER
I20260926 12:32:24.629772 25627 consensus_queue.cc:260] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.630462 25348 heartbeater.cc:503] Master 127.7.99.62:36829 was elected leader, sending a full tablet report...
I20260926 12:32:24.631747 25627 ts_tablet_manager.cc:1428] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:32:24.632335 25628 raft_consensus.cc:348] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.632586 25628 raft_consensus.cc:374] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:24.632644 25628 raft_consensus.cc:729] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667d6dfe1ad74c9bb0533ca7bf32327b, State: Initialized, Role: FOLLOWER
I20260926 12:32:24.633118 25628 consensus_queue.cc:260] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.633236 25630 raft_consensus.cc:348] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.633496 25630 raft_consensus.cc:374] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:24.633553 25630 raft_consensus.cc:729] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60ca9005a4f42b29d5274bb2fb93b03, State: Initialized, Role: FOLLOWER
I20260926 12:32:24.633935 25630 consensus_queue.cc:260] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.634063 25628 ts_tablet_manager.cc:1428] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:32:24.634411 25479 heartbeater.cc:503] Master 127.7.99.62:36829 was elected leader, sending a full tablet report...
I20260926 12:32:24.634763 25630 ts_tablet_manager.cc:1428] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:32:24.635090 25610 heartbeater.cc:503] Master 127.7.99.62:36829 was elected leader, sending a full tablet report...
I20260926 12:32:24.668500 25634 raft_consensus.cc:482] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:24.668700 25634 raft_consensus.cc:504] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.669807 25634 leader_election.cc:288] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23accffc4b6d4f36938fc0b2b9785588 (127.7.99.1:38727), e60ca9005a4f42b29d5274bb2fb93b03 (127.7.99.3:33563)
I20260926 12:32:24.674000 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.670125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:41677 (local address 127.7.99.1:38727)
0926 12:32:24.670252 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.670258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.670738 (+ 480us) server_negotiation.cc:407] Connection header received
0926 12:32:24.670851 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.670854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.670931 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.671006 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:24.671320 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.671852 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673011 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.673403 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673522 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.673585 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.673800 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.673873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:32:24.674021 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.670049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38727 (local address 127.7.99.2:41677)
0926 12:32:24.670622 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.670635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.670758 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.671033 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.671035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.671049 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:24.671183 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.671196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.671964 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.671970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.672884 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.672894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.673523 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.673670 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.673692 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.673781 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.673852 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":249,"thread_start_us":135,"threads_started":1}
I20260926 12:32:24.674046 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.670364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:32885 (local address 127.7.99.3:33563)
0926 12:32:24.670525 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.670532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.670557 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:32:24.670627 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.670630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.670683 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.670760 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:24.671166 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.671807 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673074 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.673469 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673588 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.673651 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.673855 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.673925 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 12:32:24.674046 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.670048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33563 (local address 127.7.99.2:32885)
0926 12:32:24.670164 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.670179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:24.670299 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:24.670883 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:24.670886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:24.670907 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:24.671050 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.671058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.671923 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.671928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.672952 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:24.672962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.673598 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.673602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:24.673735 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.673751 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:32:24.673841 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:32:24.673913 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":29}
I20260926 12:32:24.674642 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a5267ffd84f7a9947975a90524934" candidate_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23accffc4b6d4f36938fc0b2b9785588" is_pre_election: true
I20260926 12:32:24.674645 25564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a5267ffd84f7a9947975a90524934" candidate_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" is_pre_election: true
I20260926 12:32:24.674964 25302 raft_consensus.cc:2454] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667d6dfe1ad74c9bb0533ca7bf32327b in term 0.
I20260926 12:32:24.674964 25564 raft_consensus.cc:2454] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667d6dfe1ad74c9bb0533ca7bf32327b in term 0.
I20260926 12:32:24.675484 25368 leader_election.cc:302] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23accffc4b6d4f36938fc0b2b9785588, 667d6dfe1ad74c9bb0533ca7bf32327b, e60ca9005a4f42b29d5274bb2fb93b03; no voters:
I20260926 12:32:24.675794 25634 raft_consensus.cc:2781] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:24.675890 25634 raft_consensus.cc:482] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:24.675966 25634 raft_consensus.cc:3037] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:24.677042 25634 raft_consensus.cc:504] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.677377 25634 leader_election.cc:288] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [CANDIDATE]: Term 1 election: Requested vote from peers 23accffc4b6d4f36938fc0b2b9785588 (127.7.99.1:38727), e60ca9005a4f42b29d5274bb2fb93b03 (127.7.99.3:33563)
I20260926 12:32:24.677721 25564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a5267ffd84f7a9947975a90524934" candidate_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60ca9005a4f42b29d5274bb2fb93b03"
I20260926 12:32:24.677870 25564 raft_consensus.cc:3037] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:32:24.678472 25349 tablet.cc:2369] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:24.679067 25564 raft_consensus.cc:2454] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667d6dfe1ad74c9bb0533ca7bf32327b in term 1.
I20260926 12:32:24.679373 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a5267ffd84f7a9947975a90524934" candidate_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23accffc4b6d4f36938fc0b2b9785588"
I20260926 12:32:24.679431 25369 leader_election.cc:302] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667d6dfe1ad74c9bb0533ca7bf32327b, e60ca9005a4f42b29d5274bb2fb93b03; no voters:
I20260926 12:32:24.679508 25302 raft_consensus.cc:3037] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:24.679592 25634 raft_consensus.cc:2781] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:24.679833 25634 raft_consensus.cc:686] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 1 LEADER]: Becoming Leader. State: Replica: 667d6dfe1ad74c9bb0533ca7bf32327b, State: Running, Role: LEADER
I20260926 12:32:24.680222 25634 consensus_queue.cc:237] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } }
I20260926 12:32:24.680804 25302 raft_consensus.cc:2454] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667d6dfe1ad74c9bb0533ca7bf32327b in term 1.
I20260926 12:32:24.683743 25159 catalog_manager.cc:5495] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b reported cstate change: term changed from 0 to 1, leader changed from <none> to 667d6dfe1ad74c9bb0533ca7bf32327b (127.7.99.2). New cstate: current_term: 1 leader_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667d6dfe1ad74c9bb0533ca7bf32327b" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 41823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 } health_report { overall_health: UNKNOWN } } }
I20260926 12:32:24.694875 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.7.99.2:41823)
0926 12:32:24.691061 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:24.691068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:24.691104 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:32:24.691314 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:24.691317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:24.691370 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:24.691582 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:24.691824 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.692374 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.693398 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:24.693670 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:24.693789 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:24.694089 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:24.694493 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:24.694636 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:24.694670 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 12:32:24.694719 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:32:24.700532 25564 raft_consensus.cc:1261] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 1 FOLLOWER]: Refusing update from remote peer 667d6dfe1ad74c9bb0533ca7bf32327b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:24.700551 25302 raft_consensus.cc:1261] T 979a5267ffd84f7a9947975a90524934 P 23accffc4b6d4f36938fc0b2b9785588 [term 1 FOLLOWER]: Refusing update from remote peer 667d6dfe1ad74c9bb0533ca7bf32327b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:24.701242 25634 consensus_queue.cc:1035] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23accffc4b6d4f36938fc0b2b9785588" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:24.701416 25637 consensus_queue.cc:1035] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60ca9005a4f42b29d5274bb2fb93b03" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:24.711025 25641 mvcc.cc:205] Tried to move back new op lower bound from 7333584669486256128 to 7333584669413457920. Current Snapshot: MvccSnapshot[applied={T|T < 7333584669486256128}]
I20260926 12:32:24.718174 25160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:24.718309 25160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:24.720690 25160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01c2ff6ca3694f54a63c6c6e5a739f02: Sending DeleteTablet for 3 replicas of tablet 979a5267ffd84f7a9947975a90524934
I20260926 12:32:24.721462 25413 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a5267ffd84f7a9947975a90524934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:24 UTC) from {username='slave'} at 127.0.0.1:40954
I20260926 12:32:24.721804 25544 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a5267ffd84f7a9947975a90524934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:24 UTC) from {username='slave'} at 127.0.0.1:34658
I20260926 12:32:24.722016 25650 tablet_replica.cc:330] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: stopping tablet replica
I20260926 12:32:24.722307 25650 raft_consensus.cc:2229] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:24.722456 25282 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a5267ffd84f7a9947975a90524934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:24 UTC) from {username='slave'} at 127.0.0.1:42728
I20260926 12:32:24.722795 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25220
I20260926 12:32:24.722829 25650 raft_consensus.cc:2258] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:24.724277 25651 tablet_replica.cc:330] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: stopping tablet replica
I20260926 12:32:24.724557 25651 raft_consensus.cc:2229] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:24.724849 25651 raft_consensus.cc:2258] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:24.726204 25650 ts_tablet_manager.cc:1905] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:24.726235 25651 ts_tablet_manager.cc:1905] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:24.729077 25651 ts_tablet_manager.cc:1918] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:24.729179 25651 log.cc:1196] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/979a5267ffd84f7a9947975a90524934
I20260926 12:32:24.729475 25651 ts_tablet_manager.cc:1939] T 979a5267ffd84f7a9947975a90524934 P e60ca9005a4f42b29d5274bb2fb93b03: Deleting consensus metadata
W20260926 12:32:24.729578 25146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:38727 (error 108)
I20260926 12:32:24.730321 25650 ts_tablet_manager.cc:1918] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:24.730419 25650 log.cc:1196] T 979a5267ffd84f7a9947975a90524934 P 667d6dfe1ad74c9bb0533ca7bf32327b: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/979a5267ffd84f7a9947975a90524934
I20260926 12:32:24.730559 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25351
I20260926 12:32:24.732429 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.732067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38727 (local address 127.0.0.1:42736)
0926 12:32:24.732252 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.732329 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:32:24.732620 25146 catalog_manager.cc:4584] TS 23accffc4b6d4f36938fc0b2b9785588 (127.7.99.1:38727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 979a5267ffd84f7a9947975a90524934: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38727: connect: Connection refused (error 111)
I20260926 12:32:24.733345 25147 catalog_manager.cc:4841] TS e60ca9005a4f42b29d5274bb2fb93b03 (127.7.99.3:33563): tablet 979a5267ffd84f7a9947975a90524934 (table test-table [id=d95bfc722a104abcbe28f9c291f42a10]) successfully deleted
I20260926 12:32:24.735781 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25482
I20260926 12:32:24.740442 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:24.735833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:41823 (local address 127.0.0.1:40968)
0926 12:32:24.740322 (+ 4489us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:24.740374 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4394}
I20260926 12:32:24.741389 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25129
2026-09-26T12:32:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:24 dist-test-slave-3jw2 krb5kdc[25656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:24 dist-test-slave-3jw2 krb5kdc[25656](info): set up 2 sockets
Sep 26 12:32:24 dist-test-slave-3jw2 krb5kdc[25656](info): commencing operation
krb5kdc: starting...
W20260926 12:32:25.836294 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:25 dist-test-slave-3jw2 krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:25.958635 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41455
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:40891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:26.072422 25669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:26.072669 25669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:26.072710 25669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:26.077430 25669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:26.077486 25669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:26.077507 25669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:26.077523 25669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:26.077538 25669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:26.082682 25669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:41455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:26.083637 25669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:26.084643 25669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:32:26.092805 25669 server_base.cc:1034] running on GCE node
W20260926 12:32:26.092836 25674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.092728 25675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.092727 25677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:26.093583 25669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:26.094681 25669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:26.095889 25669 hybrid_clock.cc:648] HybridClock initialized: now 1790425946095871 us; error 31 us; skew 500 ppm
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:26.098845 25669 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:26.099660 25669 webserver.cc:460] Webserver started at http://127.7.99.62:38289/ using document root <none> and password file <none>
I20260926 12:32:26.100231 25669 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:26.100315 25669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:26.100490 25669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:26.102483 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "1faea69a34984320bc088a27859b5364"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "cdac94fb48274702b37ac432edfeac81"
server_key_iv: "810991d1993d72d6672e0c92076caf9c"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.103019 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "1faea69a34984320bc088a27859b5364"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "cdac94fb48274702b37ac432edfeac81"
server_key_iv: "810991d1993d72d6672e0c92076caf9c"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.106838 25669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 12:32:26.109215 25684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.110391 25669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:26.110514 25669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "1faea69a34984320bc088a27859b5364"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "cdac94fb48274702b37ac432edfeac81"
server_key_iv: "810991d1993d72d6672e0c92076caf9c"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.110620 25669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:26.135226 25669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:26.138342 25669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:26.138545 25669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:26.145737 25669 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:41455
I20260926 12:32:26.145752 25736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:41455 every 8 connection(s)
I20260926 12:32:26.146699 25669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:26.149600 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:26.155073 25737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: Bootstrap starting.
I20260926 12:32:26.155169 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25669
I20260926 12:32:26.155308 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:26.155594 7564 external_mini_cluster.cc:1428] Setting key e786bed1620d6d289950ee18c7d486ab
I20260926 12:32:26.157644 25737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:26.158373 25737 log.cc:824] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:26.160471 25737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: No bootstrap required, opened a new log
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:26.165107 25737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } }
I20260926 12:32:26.165360 25737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:26.165421 25737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1faea69a34984320bc088a27859b5364, State: Initialized, Role: FOLLOWER
I20260926 12:32:26.165927 25737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } }
I20260926 12:32:26.166108 25737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:26.166173 25737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:26.166289 25737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:26.167418 25737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } }
I20260926 12:32:26.167809 25737 leader_election.cc:302] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1faea69a34984320bc088a27859b5364; no voters:
I20260926 12:32:26.168130 25737 leader_election.cc:288] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:26.168386 25742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:26.168689 25742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [term 1 LEADER]: Becoming Leader. State: Replica: 1faea69a34984320bc088a27859b5364, State: Running, Role: LEADER
I20260926 12:32:26.169152 25742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } }
I20260926 12:32:26.169449 25737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:26.170918 25744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1faea69a34984320bc088a27859b5364. Latest consensus state: current_term: 1 leader_uuid: "1faea69a34984320bc088a27859b5364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } } }
I20260926 12:32:26.171046 25744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:26.171187 25743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1faea69a34984320bc088a27859b5364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faea69a34984320bc088a27859b5364" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41455 } } }
I20260926 12:32:26.171265 25743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:26.171479 25753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:26.171038 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.156982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.7.99.62:41455)
0926 12:32:26.157376 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.157391 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.157426 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:32:26.158090 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.158118 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.158449 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.158811 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:26.159619 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.160734 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.161909 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.162376 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.162493 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.165372 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:26.165381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:26.165398 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:26.165434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:26.167403 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.167990 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.167997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.168004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.168148 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.168585 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.168588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.168589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.169142 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:26.169356 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.169654 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.169964 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260926 12:32:26.174412 25753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:26.179857 25753 catalog_manager.cc:1347] Generated new cluster ID: feb08a06b87241958748d7b17765b097
I20260926 12:32:26.179920 25753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:26.186754 25753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:26.187625 25753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:26.195129 25753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: Generated new TSK 0
I20260926 12:32:26.195721 25753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:32:26.229177 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:40891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:26.335953 25763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:26.336246 25763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:26.336319 25763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:26.340284 25763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:26.340363 25763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:26.340451 25763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:26.345521 25763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:26.346529 25763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:26.347553 25763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:26.357013 25768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.357055 25771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.357056 25769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:26.357366 25763 server_base.cc:1034] running on GCE node
I20260926 12:32:26.357791 25763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:26.358453 25763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:26.359658 25763 hybrid_clock.cc:648] HybridClock initialized: now 1790425946359643 us; error 30 us; skew 500 ppm
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:26.362464 25763 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:32:26.363364 25763 webserver.cc:460] Webserver started at http://127.7.99.1:40193/ using document root <none> and password file <none>
I20260926 12:32:26.363919 25763 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:26.364009 25763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:26.364305 25763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:26.366307 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "e3b2736b1e4a4c34a53b02a5278e81a4"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "06638c0ace795121fc49806e2d731368"
server_key_iv: "e6287c9c3c560684c25ddec8be01f511"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.366884 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "e3b2736b1e4a4c34a53b02a5278e81a4"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "06638c0ace795121fc49806e2d731368"
server_key_iv: "e6287c9c3c560684c25ddec8be01f511"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.370775 25763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 12:32:26.373574 25778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.374884 25763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:26.374985 25763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "e3b2736b1e4a4c34a53b02a5278e81a4"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "06638c0ace795121fc49806e2d731368"
server_key_iv: "e6287c9c3c560684c25ddec8be01f511"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.375074 25763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:26.394413 25763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:26.397807 25763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:26.397971 25763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:26.398509 25763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:26.399374 25763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:26.399422 25763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.399464 25763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:26.399484 25763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.408623 25763 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:35511
I20260926 12:32:26.408649 25891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:35511 every 8 connection(s)
I20260926 12:32:26.409519 25763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:26.415427 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25763
I20260926 12:32:26.415580 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:26.415925 7564 external_mini_cluster.cc:1428] Setting key 2c49a620e4537b0bd663aa4407593942
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:26.424099 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.411284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:51579 (local address 127.7.99.62:41455)
0926 12:32:26.411426 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.411432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.412300 (+ 868us) server_negotiation.cc:407] Connection header received
0926 12:32:26.413552 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.413555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.413610 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.413713 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:26.414827 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.415372 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.416894 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.417309 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.417423 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.420118 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:26.420124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:26.420127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:26.420157 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:26.421978 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.422504 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.422509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.422512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.422609 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.422943 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.422949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.422953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.423210 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:26.423319 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.423767 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.423897 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:32:26.424821 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.411686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41455 (local address 127.7.99.1:51579)
0926 12:32:26.412113 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.412154 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.413347 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.413882 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.413888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.414208 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:26.414658 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.414671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.415507 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.415510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.416754 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.416765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.417466 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.417471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.417644 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.418236 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:26.418252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:26.419914 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:26.422103 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.422129 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.422140 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.422395 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.422711 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.422713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.422715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.422843 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.423322 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:26.423327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:26.423534 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.423682 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.423934 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260926 12:32:26.426028 25892 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41455
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
I20260926 12:32:26.426290 25892 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:26.426805 25892 heartbeater.cc:511] Master 127.7.99.62:41455 requested a full tablet report, sending...
I20260926 12:32:26.428399 25701 ts_manager.cc:194] Registered new tserver with Master: e3b2736b1e4a4c34a53b02a5278e81a4 (127.7.99.1:35511)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:32:26.429973 25701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:51579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:32:26.443892 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:40891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:26.553100 25897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:26.553429 25897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:26.553488 25897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:26.557493 25897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:26.557549 25897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:26.557605 25897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:26.562718 25897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:26.563731 25897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:26.564723 25897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:26.574173 25902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.574164 25905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.574371 25903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:26.574550 25897 server_base.cc:1034] running on GCE node
I20260926 12:32:26.575058 25897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:26.575731 25897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:26.576952 25897 hybrid_clock.cc:648] HybridClock initialized: now 1790425946576925 us; error 31 us; skew 500 ppm
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:26.579766 25897 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:32:26.580684 25897 webserver.cc:460] Webserver started at http://127.7.99.2:46633/ using document root <none> and password file <none>
I20260926 12:32:26.581248 25897 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:26.581339 25897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:26.581569 25897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:26.583580 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "4f4abb9779aed38da51e3e0ab7964dce"
server_key_iv: "bf8c9c998c3e8709ef01102128585347"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.584231 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "4f4abb9779aed38da51e3e0ab7964dce"
server_key_iv: "bf8c9c998c3e8709ef01102128585347"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.587765 25897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:32:26.590209 25912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.591388 25897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:32:26.591524 25897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "4f4abb9779aed38da51e3e0ab7964dce"
server_key_iv: "bf8c9c998c3e8709ef01102128585347"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.591637 25897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:26.609150 25897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:26.611887 25897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:26.612080 25897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:26.612695 25897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:26.613508 25897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:26.613579 25897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.613648 25897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:26.613691 25897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.622359 25897 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:39149
I20260926 12:32:26.622390 26025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:39149 every 8 connection(s)
I20260926 12:32:26.623202 25897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:26.629741 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 25897
I20260926 12:32:26.629885 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:26.630208 7564 external_mini_cluster.cc:1428] Setting key 656091bd5384f9a78f3414209dbc67e4
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:26.637044 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.624856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:39839 (local address 127.7.99.62:41455)
0926 12:32:26.624985 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.624989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.625787 (+ 798us) server_negotiation.cc:407] Connection header received
0926 12:32:26.627055 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.627058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.627117 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.627215 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:26.628385 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.628901 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.630351 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.630573 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.630673 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.633130 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:26.633137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:26.633143 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:26.633178 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:26.634761 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.635358 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.635360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.635361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.635435 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.635866 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.635869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.635870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.636046 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:26.636128 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.636820 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.636917 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:26.638206 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.625230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41455 (local address 127.7.99.2:39839)
0926 12:32:26.625641 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.625680 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.626841 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.627394 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.627400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.627727 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:26.628245 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.628263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.629024 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.629027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.630204 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.630217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.630738 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.630743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.630898 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.631537 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:26.631562 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:26.632965 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:26.634868 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.634876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.634893 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.635246 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.635535 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.635539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.635541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.635748 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.636170 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:26.636200 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:26.636532 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.636757 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.637119 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260926 12:32:26.639391 26026 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41455
I20260926 12:32:26.639628 26026 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:32:26.640192 26026 heartbeater.cc:511] Master 127.7.99.62:41455 requested a full tablet report, sending...
I20260926 12:32:26.641197 25701 ts_manager.cc:194] Registered new tserver with Master: 6b73a7cfb1594dd7a8d5d245ace1ef0e (127.7.99.2:39149)
I20260926 12:32:26.641772 25701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:39839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:32:26.658718 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:40891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:26.770886 26031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:26.771168 26031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:26.771211 26031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:26.775246 26031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:26.775292 26031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:26.775348 26031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:26.780458 26031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:26.781330 26031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:26.782248 26031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:26.791450 26039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.791457 26036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:26.791525 26037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:26.791689 26031 server_base.cc:1034] running on GCE node
I20260926 12:32:26.792122 26031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:26.792761 26031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:26.793934 26031 hybrid_clock.cc:648] HybridClock initialized: now 1790425946793904 us; error 45 us; skew 500 ppm
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:26.796553 26031 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:32:26.797282 26031 webserver.cc:460] Webserver started at http://127.7.99.3:35473/ using document root <none> and password file <none>
I20260926 12:32:26.797750 26031 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:26.797806 26031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:26.797956 26031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:26.799805 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "b335349f2f2744158d00fa66ddabae54"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "0f9ec53f8b626b0c526e9de0bc599444"
server_key_iv: "4e21318e5c6b2a27b6f7bae62110980a"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.800349 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "b335349f2f2744158d00fa66ddabae54"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "0f9ec53f8b626b0c526e9de0bc599444"
server_key_iv: "4e21318e5c6b2a27b6f7bae62110980a"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.803665 26031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:32:26.805873 26046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.806990 26031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:26.807102 26031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "b335349f2f2744158d00fa66ddabae54"
format_stamp: "Formatted at 2026-09-26 12:32:26 on dist-test-slave-3jw2"
server_key: "0f9ec53f8b626b0c526e9de0bc599444"
server_key_iv: "4e21318e5c6b2a27b6f7bae62110980a"
server_key_version: "encryptionkey@0"
I20260926 12:32:26.807197 26031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:26.843636 26031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:26.846567 26031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:26.846763 26031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:26.847358 26031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:26.848148 26031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:26.848212 26031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.848284 26031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:26.848323 26031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:26.857086 26031 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:46081
I20260926 12:32:26.857204 26159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:46081 every 8 connection(s)
I20260926 12:32:26.857982 26031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:26.864965 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26031
I20260926 12:32:26.865105 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:26.865414 7564 external_mini_cluster.cc:1428] Setting key 25b4ef15a14841267844b7ca9673be6e
Sep 26 12:32:26 dist-test-slave-3jw2 krb5kdc[25656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:26.872038 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.859820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:42201 (local address 127.7.99.62:41455)
0926 12:32:26.859987 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.859992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.860894 (+ 902us) server_negotiation.cc:407] Connection header received
0926 12:32:26.862140 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.862143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.862190 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.862266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:26.863764 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.864338 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.865518 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.865747 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.865875 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.868831 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:26.868834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:26.868836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:26.868861 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:26.870216 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.870696 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.870698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.870700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.870775 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.871089 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.871095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.871097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.871293 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:26.871371 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.871828 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.871919 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:32:26.872910 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.860316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41455 (local address 127.7.99.3:42201)
0926 12:32:26.860736 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.860784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.861940 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.862443 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.862452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.862845 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:26.863538 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.863556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.864450 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.864452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.865398 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.865407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.865910 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.865914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.866121 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.866767 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:26.866790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:26.868686 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:26.870324 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.870329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.870341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.870609 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.870850 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:26.870853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:26.870854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:26.870995 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:26.871394 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:26.871400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:26.871634 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.871784 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.872030 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260926 12:32:26.874068 26160 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41455
I20260926 12:32:26.874306 26160 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:26.874769 26160 heartbeater.cc:511] Master 127.7.99.62:41455 requested a full tablet report, sending...
I20260926 12:32:26.875739 25701 ts_manager.cc:194] Registered new tserver with Master: b335349f2f2744158d00fa66ddabae54 (127.7.99.3:46081)
I20260926 12:32:26.876441 25701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:42201
I20260926 12:32:26.879231 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:26.888497 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.881219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.7.99.62:41455)
0926 12:32:26.881346 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.881352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.881583 (+ 231us) server_negotiation.cc:407] Connection header received
0926 12:32:26.881633 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.881635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.881682 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.881756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:26.882405 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.882923 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.883755 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.883967 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.884050 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.885568 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:26.885571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:26.885573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:26.885596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:26.886839 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.887280 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.887282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.887283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.887355 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:26.887655 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:26.887658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:26.887659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:26.887881 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:26.888000 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.888280 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.888362 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:32:26.891381 25701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:26.894423 25701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:26.909976 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.902934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.7.99.1:35511)
0926 12:32:26.903437 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.903451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.903950 (+ 499us) server_negotiation.cc:407] Connection header received
0926 12:32:26.904037 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.904043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.904286 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.904423 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:26.906474 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.907520 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.908751 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.909344 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.909449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.909586 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.909703 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.909790 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":77,"threads_started":1}
I20260926 12:32:26.909976 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.903041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.7.99.3:46081)
0926 12:32:26.903619 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.903631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.903901 (+ 270us) server_negotiation.cc:407] Connection header received
0926 12:32:26.904129 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.904134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.904308 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.904423 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:26.905052 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.906284 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.908469 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.909172 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.909265 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.909378 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.909678 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.909790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":199,"threads_started":1}
I20260926 12:32:26.910460 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.902796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:46081 (local address 127.0.0.1:35392)
0926 12:32:26.903804 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.903817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.903994 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.904627 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.904633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.904663 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:26.904925 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.904930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.906431 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.906435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.908327 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.908339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.909294 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.909298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.909428 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.909473 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.910244 (+ 771us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.910318 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":351,"spinlock_wait_cycles":4736,"thread_start_us":107,"threads_started":1}
I20260926 12:32:26.910722 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.902796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35511 (local address 127.0.0.1:50832)
0926 12:32:26.903599 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.903649 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.903802 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.904627 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.904633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.904668 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:26.904922 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.904930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.907656 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.907663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.908629 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.908639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.909473 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.909477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.909567 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.909591 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.909670 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.909797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":49,"spinlock_wait_cycles":12288,"thread_start_us":97,"threads_started":1}
I20260926 12:32:26.910955 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.903557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:39149 (local address 127.0.0.1:56192)
0926 12:32:26.903865 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:26.903879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:26.903961 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:26.904731 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:26.904734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:26.904753 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:26.904942 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.904949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.906432 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.906435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.908327 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:26.908339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.910544 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.910547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:26.910659 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.910684 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:32:26.910751 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:32:26.910835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":322,"thread_start_us":105,"threads_started":1}
I20260926 12:32:26.911545 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:26.903722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.7.99.2:39149)
0926 12:32:26.904247 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:26.904263 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:26.904285 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:32:26.904359 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:26.904364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:26.904564 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:26.904754 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:26.905099 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.906286 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.909462 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:26.910296 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:26.910474 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:26.911220 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:26.911276 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 12:32:26.911367 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":102,"threads_started":1}
I20260926 12:32:26.913604 25826 tablet_service.cc:1467] Processing CreateTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 (DEFAULT_TABLE table=test-table [id=b372d3661fd0407499fb8d43d9f23a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:26.913703 26094 tablet_service.cc:1467] Processing CreateTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 (DEFAULT_TABLE table=test-table [id=b372d3661fd0407499fb8d43d9f23a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:26.913920 25960 tablet_service.cc:1467] Processing CreateTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 (DEFAULT_TABLE table=test-table [id=b372d3661fd0407499fb8d43d9f23a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:26.914724 25826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e375d1526e4c95b011f79a4e5058b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:26.914750 25960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e375d1526e4c95b011f79a4e5058b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:26.914757 26094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e375d1526e4c95b011f79a4e5058b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:26.920866 26177 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Bootstrap starting.
I20260926 12:32:26.921885 26178 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Bootstrap starting.
I20260926 12:32:26.922673 26179 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Bootstrap starting.
I20260926 12:32:26.923529 26177 tablet_bootstrap.cc:654] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:26.924131 26178 tablet_bootstrap.cc:654] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:26.924595 26179 tablet_bootstrap.cc:654] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:26.924896 26177 log.cc:824] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:26.925346 26178 log.cc:824] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:26.925714 26179 log.cc:824] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:26.927080 26177 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: No bootstrap required, opened a new log
I20260926 12:32:26.927300 26177 ts_tablet_manager.cc:1397] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 12:32:26.927498 26178 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: No bootstrap required, opened a new log
I20260926 12:32:26.927656 26179 tablet_bootstrap.cc:492] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: No bootstrap required, opened a new log
I20260926 12:32:26.927676 26178 ts_tablet_manager.cc:1397] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:32:26.927819 26179 ts_tablet_manager.cc:1397] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:32:26.930327 26177 raft_consensus.cc:348] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.930565 26177 raft_consensus.cc:374] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:26.930650 26177 raft_consensus.cc:729] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b2736b1e4a4c34a53b02a5278e81a4, State: Initialized, Role: FOLLOWER
I20260926 12:32:26.931118 26177 consensus_queue.cc:260] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.931510 26179 raft_consensus.cc:348] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.931510 26178 raft_consensus.cc:348] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.931775 26178 raft_consensus.cc:374] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:26.931834 26179 raft_consensus.cc:374] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:26.931869 26178 raft_consensus.cc:729] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b335349f2f2744158d00fa66ddabae54, State: Initialized, Role: FOLLOWER
I20260926 12:32:26.931938 26179 raft_consensus.cc:729] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b73a7cfb1594dd7a8d5d245ace1ef0e, State: Initialized, Role: FOLLOWER
I20260926 12:32:26.932183 26177 ts_tablet_manager.cc:1428] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 12:32:26.932379 26179 consensus_queue.cc:260] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.932379 26178 consensus_queue.cc:260] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:26.932498 25892 heartbeater.cc:503] Master 127.7.99.62:41455 was elected leader, sending a full tablet report...
I20260926 12:32:26.933197 26160 heartbeater.cc:503] Master 127.7.99.62:41455 was elected leader, sending a full tablet report...
I20260926 12:32:26.933286 26178 ts_tablet_manager.cc:1428] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:32:26.933290 26179 ts_tablet_manager.cc:1428] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:32:26.933653 26026 heartbeater.cc:503] Master 127.7.99.62:41455 was elected leader, sending a full tablet report...
W20260926 12:32:27.108946 26161 tablet.cc:2369] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:32:27.124420 26027 tablet.cc:2369] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:32:27.160971 25893 tablet.cc:2369] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:27.313468 26185 raft_consensus.cc:482] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:27.313685 26185 raft_consensus.cc:504] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:27.314869 26185 leader_election.cc:288] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b2736b1e4a4c34a53b02a5278e81a4 (127.7.99.1:35511), b335349f2f2744158d00fa66ddabae54 (127.7.99.3:46081)
I20260926 12:32:27.320068 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.315259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:60597 (local address 127.7.99.3:46081)
0926 12:32:27.315386 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:27.315391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:27.315656 (+ 265us) server_negotiation.cc:407] Connection header received
0926 12:32:27.315809 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:27.315812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:27.315863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:27.315952 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:27.316285 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.316992 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.318828 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.319424 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.319531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.319596 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:27.319866 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 12:32:27.319934 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:32:27.320068 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.315236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:46757 (local address 127.7.99.1:35511)
0926 12:32:27.315388 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:27.315393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:27.315567 (+ 174us) server_negotiation.cc:407] Connection header received
0926 12:32:27.315734 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:27.315737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:27.315788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:27.315870 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:27.316144 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.316992 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.318826 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.319424 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.319531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.319597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:27.319865 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:32:27.319934 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:32:27.320132 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.315111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:46081 (local address 127.7.99.2:60597)
0926 12:32:27.315540 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:27.315552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:27.315694 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:27.315960 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:27.315963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:27.315978 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:27.316126 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.316132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.317148 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.317150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.318701 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.318714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.319547 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.319550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.319714 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.319756 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 12:32:27.319836 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:32:27.319937 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":357,"thread_start_us":52,"threads_started":1}
I20260926 12:32:27.320122 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.315111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35511 (local address 127.7.99.2:46757)
0926 12:32:27.315476 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:27.315489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:27.315625 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:27.315881 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:27.315884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:27.315900 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:27.316030 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.316036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.317117 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.317120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.318712 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.318718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.319547 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.319550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.319730 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.319768 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:32:27.319836 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:32:27.319937 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":188,"thread_start_us":74,"threads_started":1}
I20260926 12:32:27.320966 25846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e375d1526e4c95b011f79a4e5058b9" candidate_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" is_pre_election: true
I20260926 12:32:27.321195 26114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e375d1526e4c95b011f79a4e5058b9" candidate_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b335349f2f2744158d00fa66ddabae54" is_pre_election: true
I20260926 12:32:27.321321 25846 raft_consensus.cc:2454] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b73a7cfb1594dd7a8d5d245ace1ef0e in term 0.
I20260926 12:32:27.321471 26114 raft_consensus.cc:2454] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b73a7cfb1594dd7a8d5d245ace1ef0e in term 0.
I20260926 12:32:27.321843 25916 leader_election.cc:302] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b73a7cfb1594dd7a8d5d245ace1ef0e, e3b2736b1e4a4c34a53b02a5278e81a4; no voters:
I20260926 12:32:27.322136 26185 raft_consensus.cc:2781] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:27.322213 26185 raft_consensus.cc:482] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:27.322263 26185 raft_consensus.cc:3037] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:27.323280 26185 raft_consensus.cc:504] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:27.323642 26185 leader_election.cc:288] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [CANDIDATE]: Term 1 election: Requested vote from peers e3b2736b1e4a4c34a53b02a5278e81a4 (127.7.99.1:35511), b335349f2f2744158d00fa66ddabae54 (127.7.99.3:46081)
I20260926 12:32:27.323951 25846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e375d1526e4c95b011f79a4e5058b9" candidate_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4"
I20260926 12:32:27.323944 26114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e375d1526e4c95b011f79a4e5058b9" candidate_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b335349f2f2744158d00fa66ddabae54"
I20260926 12:32:27.324069 26114 raft_consensus.cc:3037] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:27.324070 25846 raft_consensus.cc:3037] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:27.325376 26114 raft_consensus.cc:2454] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b73a7cfb1594dd7a8d5d245ace1ef0e in term 1.
I20260926 12:32:27.325376 25846 raft_consensus.cc:2454] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b73a7cfb1594dd7a8d5d245ace1ef0e in term 1.
I20260926 12:32:27.325673 25915 leader_election.cc:302] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b73a7cfb1594dd7a8d5d245ace1ef0e, b335349f2f2744158d00fa66ddabae54; no voters:
I20260926 12:32:27.325834 26185 raft_consensus.cc:2781] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:27.326045 26185 raft_consensus.cc:686] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 1 LEADER]: Becoming Leader. State: Replica: 6b73a7cfb1594dd7a8d5d245ace1ef0e, State: Running, Role: LEADER
I20260926 12:32:27.326398 26185 consensus_queue.cc:237] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } }
I20260926 12:32:27.329522 25699 catalog_manager.cc:5495] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b73a7cfb1594dd7a8d5d245ace1ef0e (127.7.99.2). New cstate: current_term: 1 leader_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b73a7cfb1594dd7a8d5d245ace1ef0e" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39149 } health_report { overall_health: HEALTHY } } }
I20260926 12:32:27.368426 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.365001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.7.99.2:39149)
0926 12:32:27.365158 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:27.365162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:27.365178 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:32:27.365254 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:27.365256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:27.365315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:27.365422 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:27.365647 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.366225 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.367048 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.367277 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.367366 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.367623 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:27.368033 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:27.368219 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:27.368280 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:32:27.368313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:32:27.374646 26114 raft_consensus.cc:1261] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 1 FOLLOWER]: Refusing update from remote peer 6b73a7cfb1594dd7a8d5d245ace1ef0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:27.374650 25846 raft_consensus.cc:1261] T 34e375d1526e4c95b011f79a4e5058b9 P e3b2736b1e4a4c34a53b02a5278e81a4 [term 1 FOLLOWER]: Refusing update from remote peer 6b73a7cfb1594dd7a8d5d245ace1ef0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:27.375358 26188 consensus_queue.cc:1035] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b335349f2f2744158d00fa66ddabae54" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 46081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:27.375511 26185 consensus_queue.cc:1035] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b2736b1e4a4c34a53b02a5278e81a4" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:27.393652 25701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:27.393826 25701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:27.396034 25701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1faea69a34984320bc088a27859b5364: Sending DeleteTablet for 3 replicas of tablet 34e375d1526e4c95b011f79a4e5058b9
I20260926 12:32:27.396845 26094 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:35392
I20260926 12:32:27.396873 25826 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:50832
I20260926 12:32:27.397439 25960 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e375d1526e4c95b011f79a4e5058b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:27 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:56192
I20260926 12:32:27.397517 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25763
I20260926 12:32:27.397562 26201 tablet_replica.cc:330] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: stopping tablet replica
I20260926 12:32:27.397897 26201 raft_consensus.cc:2229] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:27.398214 26201 raft_consensus.cc:2258] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:27.400226 26201 ts_tablet_manager.cc:1905] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:27.400378 26203 tablet_replica.cc:330] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: stopping tablet replica
I20260926 12:32:27.400707 26203 raft_consensus.cc:2229] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:27.401166 26203 raft_consensus.cc:2258] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:27.402675 26203 ts_tablet_manager.cc:1905] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:27.403113 26201 ts_tablet_manager.cc:1918] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:27.403267 26201 log.cc:1196] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/34e375d1526e4c95b011f79a4e5058b9
I20260926 12:32:27.403680 26201 ts_tablet_manager.cc:1939] T 34e375d1526e4c95b011f79a4e5058b9 P b335349f2f2744158d00fa66ddabae54: Deleting consensus metadata
I20260926 12:32:27.405103 25687 catalog_manager.cc:4841] TS b335349f2f2744158d00fa66ddabae54 (127.7.99.3:46081): tablet 34e375d1526e4c95b011f79a4e5058b9 (table test-table [id=b372d3661fd0407499fb8d43d9f23a28]) successfully deleted
I20260926 12:32:27.405532 26203 ts_tablet_manager.cc:1918] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:27.405642 26203 log.cc:1196] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/34e375d1526e4c95b011f79a4e5058b9
I20260926 12:32:27.405929 26203 ts_tablet_manager.cc:1939] T 34e375d1526e4c95b011f79a4e5058b9 P 6b73a7cfb1594dd7a8d5d245ace1ef0e: Deleting consensus metadata
W20260926 12:32:27.406761 25688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:35511 (error 108)
I20260926 12:32:27.407253 25688 catalog_manager.cc:4841] TS 6b73a7cfb1594dd7a8d5d245ace1ef0e (127.7.99.2:39149): tablet 34e375d1526e4c95b011f79a4e5058b9 (table test-table [id=b372d3661fd0407499fb8d43d9f23a28]) successfully deleted
W20260926 12:32:27.407824 26172 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:27.407565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:35511 (local address 127.0.0.1:50842)
0926 12:32:27.407687 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:27.407744 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35511: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:32:27.408004 25688 catalog_manager.cc:4584] TS e3b2736b1e4a4c34a53b02a5278e81a4 (127.7.99.1:35511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34e375d1526e4c95b011f79a4e5058b9: Network error: Client connection negotiation failed: client connection to 127.7.99.1:35511: connect: Connection reset by peer (error 104)
I20260926 12:32:27.408037 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25897
I20260926 12:32:27.413492 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26031
I20260926 12:32:27.418758 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 25669
2026-09-26T12:32:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2678 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T12:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:27Z Disabled control of system clock
I20260926 12:32:27.449965 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41181
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:27.560949 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:27.561213 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:27.561249 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:27.565234 26208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:27.565284 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:27.565299 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:27.565315 26208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:27.565330 26208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:27.570020 26208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:41181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:27.570964 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:27.572263 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:27.580550 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:27.580546 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:27.580550 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:27.581063 26208 server_base.cc:1034] running on GCE node
I20260926 12:32:27.581637 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:27.582765 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:27.583940 26208 hybrid_clock.cc:648] HybridClock initialized: now 1790425947583912 us; error 43 us; skew 500 ppm
I20260926 12:32:27.586055 26208 webserver.cc:460] Webserver started at http://127.7.99.62:33491/ using document root <none> and password file <none>
I20260926 12:32:27.586701 26208 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:27.586803 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:27.587157 26208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:27.589447 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f1dc749948ce49449791ae3be99664dc"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "0e424030e0ad8d25933086bd370828db"
server_key_iv: "63d3ec02765dc68924db999bb6b94d55"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.590077 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f1dc749948ce49449791ae3be99664dc"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "0e424030e0ad8d25933086bd370828db"
server_key_iv: "63d3ec02765dc68924db999bb6b94d55"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.593883 26208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:32:27.596417 26222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:27.597620 26208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:32:27.597766 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f1dc749948ce49449791ae3be99664dc"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "0e424030e0ad8d25933086bd370828db"
server_key_iv: "63d3ec02765dc68924db999bb6b94d55"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.597882 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:27.612461 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:27.613093 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:27.613287 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:27.619974 26208 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:41181
I20260926 12:32:27.619980 26274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:41181 every 8 connection(s)
I20260926 12:32:27.620887 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:27.623816 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:27.625664 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26208
I20260926 12:32:27.625849 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:27.626145 7564 external_mini_cluster.cc:1428] Setting key 24686a1aca87a70fb91aac971d2202f1
I20260926 12:32:27.629660 26275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: Bootstrap starting.
I20260926 12:32:27.631814 26275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:27.632581 26275 log.cc:824] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:27.634690 26275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: No bootstrap required, opened a new log
I20260926 12:32:27.635648 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.627490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.7.99.62:41181)
0926 12:32:27.628069 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:27.628083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:27.628126 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:32:27.628891 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:27.628911 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:27.629320 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:27.629665 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:27.630189 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.631183 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.632126 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.632670 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.632833 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.633709 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:27.633715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:27.633729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:27.634202 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:27.634269 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:27.634295 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:27.634392 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:27.634507 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 12:32:27.634777 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260926 12:32:27.638605 26275 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } }
I20260926 12:32:27.638808 26275 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:27.638840 26275 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dc749948ce49449791ae3be99664dc, State: Initialized, Role: FOLLOWER
I20260926 12:32:27.639179 26275 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } }
I20260926 12:32:27.639281 26275 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:27.639317 26275 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:27.639382 26275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:27.640196 26275 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } }
I20260926 12:32:27.640537 26275 leader_election.cc:302] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1dc749948ce49449791ae3be99664dc; no voters:
I20260926 12:32:27.640890 26275 leader_election.cc:288] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:27.640985 26280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:27.641191 26280 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [term 1 LEADER]: Becoming Leader. State: Replica: f1dc749948ce49449791ae3be99664dc, State: Running, Role: LEADER
I20260926 12:32:27.641469 26280 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } }
I20260926 12:32:27.642016 26275 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:27.643055 26281 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1dc749948ce49449791ae3be99664dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } } }
I20260926 12:32:27.643190 26281 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:27.643447 26282 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1dc749948ce49449791ae3be99664dc. Latest consensus state: current_term: 1 leader_uuid: "f1dc749948ce49449791ae3be99664dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc749948ce49449791ae3be99664dc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41181 } } }
I20260926 12:32:27.643528 26282 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc [sys.catalog]: This master's current role is: LEADER
W20260926 12:32:27.646795 26295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:27.646857 26295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:27.646931 26289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:27.647588 26289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:27.653057 26289 catalog_manager.cc:1347] Generated new cluster ID: 08db1f642d46436297ca4dcd21bbee45
I20260926 12:32:27.653121 26289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:27.664294 26289 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:27.665190 26289 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:27.675588 26289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: Generated new TSK 0
I20260926 12:32:27.676348 26289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:27.682590 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--builtin_ntp_servers=127.7.99.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:27.812866 26299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:27.813115 26299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:27.813153 26299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:27.817113 26299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:27.817157 26299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:27.817207 26299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:27.821887 26299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:27.822819 26299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:27.824577 26299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:27.834157 26307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:27.834087 26304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:27.834066 26305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:27.834198 26299 server_base.cc:1034] running on GCE node
I20260926 12:32:27.834744 26299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:27.835350 26299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:27.836527 26299 hybrid_clock.cc:648] HybridClock initialized: now 1790425947836511 us; error 33 us; skew 500 ppm
I20260926 12:32:27.838327 26299 webserver.cc:460] Webserver started at http://127.7.99.1:41841/ using document root <none> and password file <none>
I20260926 12:32:27.838922 26299 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:27.839010 26299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:27.839244 26299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:27.841284 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "782bbbb3668d4ac6955d3c2bbf590351"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "cb5584f61540acd0a516a16a6c85887d"
server_key_iv: "9dc581433bf7699e1ebd41b2d460833d"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.841826 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "782bbbb3668d4ac6955d3c2bbf590351"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "cb5584f61540acd0a516a16a6c85887d"
server_key_iv: "9dc581433bf7699e1ebd41b2d460833d"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.845522 26299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:32:27.847846 26313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:27.849085 26299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:27.849222 26299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "782bbbb3668d4ac6955d3c2bbf590351"
format_stamp: "Formatted at 2026-09-26 12:32:27 on dist-test-slave-3jw2"
server_key: "cb5584f61540acd0a516a16a6c85887d"
server_key_iv: "9dc581433bf7699e1ebd41b2d460833d"
server_key_version: "encryptionkey@0"
I20260926 12:32:27.849344 26299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:27.873447 26299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:27.874130 26299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:27.874334 26299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:27.874917 26299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:27.875835 26299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:27.875909 26299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:27.876003 26299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:27.876056 26299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:27.885272 26299 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:36747
I20260926 12:32:27.885293 26426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:36747 every 8 connection(s)
I20260926 12:32:27.886142 26299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:27.889055 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26299
I20260926 12:32:27.889170 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:27.889441 7564 external_mini_cluster.cc:1428] Setting key e17faedc3f6a86fa8f3c8b4046afa257
I20260926 12:32:27.891479 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--builtin_ntp_servers=127.7.99.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:27.896689 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.887935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:53749 (local address 127.7.99.62:41181)
0926 12:32:27.888070 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:27.888074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:27.889134 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 12:32:27.890750 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:27.890753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:27.890804 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:27.890880 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:27.891866 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.892760 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.894426 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.894850 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.894972 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.896052 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:27.896059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:27.896064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:27.896081 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:27.896109 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:27.896123 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:27.896254 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:27.896401 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 12:32:27.896477 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:32:27.897418 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:27.888345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41181 (local address 127.7.99.1:53749)
0926 12:32:27.888981 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:27.889019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:27.890518 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:27.891060 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:27.891067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:27.891151 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:27.891722 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.891737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.892911 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.892922 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.894307 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:27.894316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:27.894991 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:27.894993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:27.895127 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:27.895793 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:27.895795 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:27.895834 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:27.895894 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:27.896265 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:27.896272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:27.896276 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:32:27.896368 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 12:32:27.896581 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260926 12:32:27.898630 26427 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41181
I20260926 12:32:27.898890 26427 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:27.899546 26427 heartbeater.cc:511] Master 127.7.99.62:41181 requested a full tablet report, sending...
I20260926 12:32:27.901162 26239 ts_manager.cc:194] Registered new tserver with Master: 782bbbb3668d4ac6955d3c2bbf590351 (127.7.99.1:36747)
I20260926 12:32:27.902364 26239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.1:53749
W20260926 12:32:28.001821 26430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:28.002120 26430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:28.002197 26430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:28.006292 26430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:28.006356 26430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:28.006459 26430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:28.011283 26430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:28.012331 26430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:28.013645 26430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:28.022748 26436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:28.022907 26430 server_base.cc:1034] running on GCE node
W20260926 12:32:28.022763 26438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:28.022818 26435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:28.023401 26430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:28.024037 26430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:28.025218 26430 hybrid_clock.cc:648] HybridClock initialized: now 1790425948025183 us; error 45 us; skew 500 ppm
I20260926 12:32:28.027025 26430 webserver.cc:460] Webserver started at http://127.7.99.2:46781/ using document root <none> and password file <none>
I20260926 12:32:28.027621 26430 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:28.027719 26430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:28.027940 26430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:28.029970 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "628a26b48977456990b5715ade47a456"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "f13645b610627d2a9215033cce278540"
server_key_iv: "c7492a91a6ab4b187122acee01b4cf8d"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.030519 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "628a26b48977456990b5715ade47a456"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "f13645b610627d2a9215033cce278540"
server_key_iv: "c7492a91a6ab4b187122acee01b4cf8d"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.034016 26430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:28.036288 26444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.037573 26430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:28.037700 26430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "628a26b48977456990b5715ade47a456"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "f13645b610627d2a9215033cce278540"
server_key_iv: "c7492a91a6ab4b187122acee01b4cf8d"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.037828 26430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:28.056986 26430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:28.057610 26430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:28.057797 26430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:28.058360 26430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:28.059257 26430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:28.059326 26430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.059397 26430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:28.059439 26430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.068095 26430 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:40505
I20260926 12:32:28.068125 26557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:40505 every 8 connection(s)
I20260926 12:32:28.068979 26430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:28.077347 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.070654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:48759 (local address 127.7.99.62:41181)
0926 12:32:28.070822 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.070827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.071820 (+ 993us) server_negotiation.cc:407] Connection header received
0926 12:32:28.073120 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.073122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.073168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.073247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:28.074064 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.074593 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.075665 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.075873 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.075951 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.076947 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:28.076950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:28.076952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:28.076963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:28.076981 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:28.076989 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:28.077046 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.077174 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.077232 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:32:28.078104 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.071050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41181 (local address 127.7.99.2:48759)
0926 12:32:28.071668 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.071706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.072932 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.073397 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.073404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.073485 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:28.073949 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.073959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.074696 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.074699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.075556 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.075564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.076001 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.076003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.076133 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.076720 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:28.076721 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:28.076758 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:28.076817 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:28.077060 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:28.077066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:28.077070 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.077158 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.077362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260926 12:32:28.078388 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26430
I20260926 12:32:28.078510 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:28.078898 7564 external_mini_cluster.cc:1428] Setting key db1c6f9c3a485700b83f2916e40daf6a
I20260926 12:32:28.079319 26558 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41181
I20260926 12:32:28.079546 26558 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:28.080118 26558 heartbeater.cc:511] Master 127.7.99.62:41181 requested a full tablet report, sending...
I20260926 12:32:28.080817 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--builtin_ntp_servers=127.7.99.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:28.081125 26239 ts_manager.cc:194] Registered new tserver with Master: 628a26b48977456990b5715ade47a456 (127.7.99.2:40505)
I20260926 12:32:28.081666 26239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.2:48759
W20260926 12:32:28.189023 26561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:28.189271 26561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:28.189309 26561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:28.193267 26561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:28.193315 26561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:28.193367 26561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:28.197978 26561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:28.198889 26561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:28.200218 26561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:28.209548 26566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:28.209551 26567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:28.209615 26561 server_base.cc:1034] running on GCE node
W20260926 12:32:28.209561 26569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:28.210307 26561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:28.210860 26561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:28.212044 26561 hybrid_clock.cc:648] HybridClock initialized: now 1790425948212029 us; error 23 us; skew 500 ppm
I20260926 12:32:28.213872 26561 webserver.cc:460] Webserver started at http://127.7.99.3:39013/ using document root <none> and password file <none>
I20260926 12:32:28.214444 26561 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:28.214515 26561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:28.214741 26561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:28.216897 26561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "a8f8f205573e49e582dbed3681f35660"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "d576616e31c0cfdb301ea9a5b4bd642d"
server_key_iv: "12d5d19c93eff7d5826261e48a1709f9"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.217459 26561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "a8f8f205573e49e582dbed3681f35660"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "d576616e31c0cfdb301ea9a5b4bd642d"
server_key_iv: "12d5d19c93eff7d5826261e48a1709f9"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.221047 26561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:32:28.223479 26575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.224853 26561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:28.224989 26561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "a8f8f205573e49e582dbed3681f35660"
format_stamp: "Formatted at 2026-09-26 12:32:28 on dist-test-slave-3jw2"
server_key: "d576616e31c0cfdb301ea9a5b4bd642d"
server_key_iv: "12d5d19c93eff7d5826261e48a1709f9"
server_key_version: "encryptionkey@0"
I20260926 12:32:28.225112 26561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:28.257277 26561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:28.258030 26561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:28.258240 26561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:28.258826 26561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:28.259781 26561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:28.259881 26561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.259956 26561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:28.260008 26561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:28.268702 26561 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:33513
I20260926 12:32:28.268728 26688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:33513 every 8 connection(s)
I20260926 12:32:28.269573 26561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:28.276784 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26561
I20260926 12:32:28.276907 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:28.277243 7564 external_mini_cluster.cc:1428] Setting key ff5c4b441beae5f11a34838f9e974e07
I20260926 12:32:28.278894 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.271395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:58915 (local address 127.7.99.62:41181)
0926 12:32:28.271543 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.271546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.272614 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 12:32:28.273934 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.273936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.273982 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.274057 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:28.274904 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.275526 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.276701 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.277087 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.277218 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.278362 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:28.278366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:28.278369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:28.278383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:28.278405 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:28.278416 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:28.278488 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.278669 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.278741 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:32:28.279780 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.271789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:41181 (local address 127.7.99.3:58915)
0926 12:32:28.272484 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.272525 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.273742 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.274216 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.274222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.274311 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:28.274771 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.274784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.275638 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.275641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.276578 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.276588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.277252 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.277263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.277453 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.278074 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:28.278076 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:28.278114 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:28.278175 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:28.278509 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:28.278518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:28.278523 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.278652 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.278933 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260926 12:32:28.280874 26689 heartbeater.cc:347] Connected to a master server at 127.7.99.62:41181
I20260926 12:32:28.281054 26689 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:28.281482 26689 heartbeater.cc:511] Master 127.7.99.62:41181 requested a full tablet report, sending...
I20260926 12:32:28.282382 26239 ts_manager.cc:194] Registered new tserver with Master: a8f8f205573e49e582dbed3681f35660 (127.7.99.3:33513)
I20260926 12:32:28.282886 26239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.3:58915
I20260926 12:32:28.291214 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:28.296684 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.293156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.7.99.62:41181)
0926 12:32:28.293283 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.293293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.293478 (+ 185us) server_negotiation.cc:407] Connection header received
0926 12:32:28.293545 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.293547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.293593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.293665 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:28.293918 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.294447 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.295200 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.295402 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.295476 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.296278 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:28.296286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:28.296289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:28.296299 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:28.296317 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:28.296325 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:28.296424 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.296544 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.296577 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:32:28.299639 26239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:28.302644 26239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:28.316911 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.310665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40505 (local address 127.0.0.1:46298)
0926 12:32:28.311051 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.311097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.311211 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.312443 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.312451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.312493 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:28.312706 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.312713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.313790 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.313796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.315354 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.315365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.316322 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.316329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.316460 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.316498 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.316611 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.316731 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":105,"spinlock_wait_cycles":3712,"thread_start_us":77,"threads_started":1}
I20260926 12:32:28.316973 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.311446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.7.99.2:40505)
0926 12:32:28.311865 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.311884 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.311926 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:32:28.312005 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.312010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.312247 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.312447 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:28.312851 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.313683 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.315499 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.316172 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.316303 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.316491 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.316656 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.316775 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":108,"threads_started":1}
I20260926 12:32:28.317247 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.311499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.7.99.3:33513)
0926 12:32:28.312022 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.312033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.312052 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:32:28.312113 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.312117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.312294 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.312661 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:28.312812 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.313912 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.315482 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.316211 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.316453 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.316637 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.316975 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.317076 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":147,"threads_started":1}
I20260926 12:32:28.317337 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.310696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:33513 (local address 127.0.0.1:34856)
0926 12:32:28.311496 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.311508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.311596 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.312446 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.312450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.312489 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:28.312705 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.312713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.314029 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.314032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.315358 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.315365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.316358 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.316361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.316712 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.316732 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.316816 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.316871 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":428,"spinlock_wait_cycles":10240,"thread_start_us":83,"threads_started":1}
I20260926 12:32:28.317727 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.310908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:36747 (local address 127.0.0.1:46026)
0926 12:32:28.312696 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.312709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.312801 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.313475 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.313478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.313496 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:28.313570 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.313576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.314630 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.314636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.315768 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.315779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.317450 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.317453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.317528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.317545 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.317595 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.317639 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 12:32:28.318081 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.311477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.7.99.1:36747)
0926 12:32:28.313051 (+ 1574us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.313062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.313085 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:32:28.313154 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.313159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.313330 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.313474 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:28.313686 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.314512 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.315905 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.316705 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.316803 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.316969 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.317798 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.317892 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":99,"threads_started":1}
I20260926 12:32:28.320122 26492 tablet_service.cc:1467] Processing CreateTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 (DEFAULT_TABLE table=test-table [id=42053cc5785f45edab6fdc29d6c3518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:28.320430 26361 tablet_service.cc:1467] Processing CreateTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 (DEFAULT_TABLE table=test-table [id=42053cc5785f45edab6fdc29d6c3518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:28.321162 26492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ab6fad1c14fa68edb9a5cbca6de16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:28.321403 26361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ab6fad1c14fa68edb9a5cbca6de16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:28.327965 26707 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Bootstrap starting.
I20260926 12:32:28.328612 26706 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Bootstrap starting.
I20260926 12:32:28.329075 26623 tablet_service.cc:1467] Processing CreateTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 (DEFAULT_TABLE table=test-table [id=42053cc5785f45edab6fdc29d6c3518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:28.330163 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ab6fad1c14fa68edb9a5cbca6de16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:28.330274 26707 tablet_bootstrap.cc:654] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:28.330569 26706 tablet_bootstrap.cc:654] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:28.331122 26707 log.cc:824] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:28.331248 26706 log.cc:824] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:28.333243 26707 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: No bootstrap required, opened a new log
I20260926 12:32:28.333243 26706 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: No bootstrap required, opened a new log
I20260926 12:32:28.333446 26707 ts_tablet_manager.cc:1397] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:32:28.333477 26706 ts_tablet_manager.cc:1397] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:32:28.336417 26710 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Bootstrap starting.
I20260926 12:32:28.337775 26706 raft_consensus.cc:348] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.337800 26707 raft_consensus.cc:348] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.338037 26707 raft_consensus.cc:374] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:28.338038 26706 raft_consensus.cc:374] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:28.338094 26706 raft_consensus.cc:729] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782bbbb3668d4ac6955d3c2bbf590351, State: Initialized, Role: FOLLOWER
I20260926 12:32:28.338094 26707 raft_consensus.cc:729] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628a26b48977456990b5715ade47a456, State: Initialized, Role: FOLLOWER
I20260926 12:32:28.338569 26710 tablet_bootstrap.cc:654] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:28.338546 26707 consensus_queue.cc:260] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.338546 26706 consensus_queue.cc:260] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.339418 26558 heartbeater.cc:503] Master 127.7.99.62:41181 was elected leader, sending a full tablet report...
I20260926 12:32:28.339514 26706 ts_tablet_manager.cc:1428] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:32:28.339515 26707 ts_tablet_manager.cc:1428] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:32:28.339897 26710 log.cc:824] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:28.340590 26427 heartbeater.cc:503] Master 127.7.99.62:41181 was elected leader, sending a full tablet report...
I20260926 12:32:28.342157 26710 tablet_bootstrap.cc:492] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: No bootstrap required, opened a new log
I20260926 12:32:28.342372 26710 ts_tablet_manager.cc:1397] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 12:32:28.346021 26710 raft_consensus.cc:348] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.346246 26710 raft_consensus.cc:374] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:28.346305 26710 raft_consensus.cc:729] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f8f205573e49e582dbed3681f35660, State: Initialized, Role: FOLLOWER
I20260926 12:32:28.346678 26710 consensus_queue.cc:260] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.347342 26689 heartbeater.cc:503] Master 127.7.99.62:41181 was elected leader, sending a full tablet report...
I20260926 12:32:28.347581 26710 ts_tablet_manager.cc:1428] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260926 12:32:28.387315 26428 tablet.cc:2369] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:28.473656 26714 raft_consensus.cc:482] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:28.473848 26714 raft_consensus.cc:504] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.475370 26714 leader_election.cc:288] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 628a26b48977456990b5715ade47a456 (127.7.99.2:40505), 782bbbb3668d4ac6955d3c2bbf590351 (127.7.99.1:36747)
I20260926 12:32:28.479156 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.475969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:39247 (local address 127.7.99.1:36747)
0926 12:32:28.476101 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.476106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.476122 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:32:28.476243 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.476247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.476293 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.476376 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:28.476667 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.477194 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478222 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.478605 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478733 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.478793 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.478970 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.479035 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:32:28.479233 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.475600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:36747 (local address 127.7.99.3:39247)
0926 12:32:28.475981 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.475995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.476112 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.476378 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.476380 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.476416 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:28.476544 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.476550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.477305 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.477307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.478082 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.478092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.478735 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.478836 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.478865 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.478954 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.479019 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":26,"thread_start_us":124,"threads_started":1}
I20260926 12:32:28.479250 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.475754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:57581 (local address 127.7.99.2:40505)
0926 12:32:28.475912 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.475916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.475933 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:32:28.475992 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.475996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.476057 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.476141 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:28.476458 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.477049 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478118 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.478645 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.478799 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.479071 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.479148 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 12:32:28.479245 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.475570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:40505 (local address 127.7.99.3:57581)
0926 12:32:28.475707 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.475724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:28.475821 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:28.476188 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:28.476193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:28.476218 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:28.476338 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.476348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.477191 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.477194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.477995 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:28.478003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.478873 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.478876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:28.478948 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.478964 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:32:28.479055 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:32:28.479120 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260926 12:32:28.479903 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ab6fad1c14fa68edb9a5cbca6de16" candidate_uuid: "a8f8f205573e49e582dbed3681f35660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782bbbb3668d4ac6955d3c2bbf590351" is_pre_election: true
I20260926 12:32:28.480239 26381 raft_consensus.cc:2454] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f8f205573e49e582dbed3681f35660 in term 0.
I20260926 12:32:28.480273 26512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ab6fad1c14fa68edb9a5cbca6de16" candidate_uuid: "a8f8f205573e49e582dbed3681f35660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628a26b48977456990b5715ade47a456" is_pre_election: true
I20260926 12:32:28.480638 26512 raft_consensus.cc:2454] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f8f205573e49e582dbed3681f35660 in term 0.
I20260926 12:32:28.480787 26578 leader_election.cc:302] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782bbbb3668d4ac6955d3c2bbf590351, a8f8f205573e49e582dbed3681f35660; no voters:
I20260926 12:32:28.481014 26714 raft_consensus.cc:2781] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:28.481098 26714 raft_consensus.cc:482] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:28.481165 26714 raft_consensus.cc:3037] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:28.482137 26714 raft_consensus.cc:504] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.482508 26714 leader_election.cc:288] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [CANDIDATE]: Term 1 election: Requested vote from peers 628a26b48977456990b5715ade47a456 (127.7.99.2:40505), 782bbbb3668d4ac6955d3c2bbf590351 (127.7.99.1:36747)
I20260926 12:32:28.482841 26512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ab6fad1c14fa68edb9a5cbca6de16" candidate_uuid: "a8f8f205573e49e582dbed3681f35660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628a26b48977456990b5715ade47a456"
I20260926 12:32:28.482844 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ab6fad1c14fa68edb9a5cbca6de16" candidate_uuid: "a8f8f205573e49e582dbed3681f35660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782bbbb3668d4ac6955d3c2bbf590351"
I20260926 12:32:28.482973 26512 raft_consensus.cc:3037] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:28.482981 26381 raft_consensus.cc:3037] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:28.483990 26381 raft_consensus.cc:2454] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f8f205573e49e582dbed3681f35660 in term 1.
I20260926 12:32:28.483999 26512 raft_consensus.cc:2454] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f8f205573e49e582dbed3681f35660 in term 1.
I20260926 12:32:28.484290 26578 leader_election.cc:302] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782bbbb3668d4ac6955d3c2bbf590351, a8f8f205573e49e582dbed3681f35660; no voters:
I20260926 12:32:28.484452 26714 raft_consensus.cc:2781] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:28.484686 26714 raft_consensus.cc:686] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 1 LEADER]: Becoming Leader. State: Replica: a8f8f205573e49e582dbed3681f35660, State: Running, Role: LEADER
I20260926 12:32:28.484993 26714 consensus_queue.cc:237] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } }
I20260926 12:32:28.488277 26239 catalog_manager.cc:5495] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8f8f205573e49e582dbed3681f35660 (127.7.99.3). New cstate: current_term: 1 leader_uuid: "a8f8f205573e49e582dbed3681f35660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f8f205573e49e582dbed3681f35660" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 33513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20260926 12:32:28.506644 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.502972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.7.99.3:33513)
0926 12:32:28.503126 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:28.503132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:28.503146 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:32:28.503349 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:28.503351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:28.503406 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:28.503518 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:28.503767 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.504343 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.505174 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:28.505398 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:28.505502 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:28.505809 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:28.506207 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:28.506361 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:28.506447 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 12:32:28.506480 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:32:28.512629 26512 raft_consensus.cc:1261] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 1 FOLLOWER]: Refusing update from remote peer a8f8f205573e49e582dbed3681f35660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:28.512637 26381 raft_consensus.cc:1261] T 071ab6fad1c14fa68edb9a5cbca6de16 P 782bbbb3668d4ac6955d3c2bbf590351 [term 1 FOLLOWER]: Refusing update from remote peer a8f8f205573e49e582dbed3681f35660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:28.513379 26716 consensus_queue.cc:1035] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628a26b48977456990b5715ade47a456" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 40505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:28.513574 26714 consensus_queue.cc:1035] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782bbbb3668d4ac6955d3c2bbf590351" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:32:28.520462 26690 tablet.cc:2369] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:28.521251 26719 mvcc.cc:205] Tried to move back new op lower bound from 7333584685100064768 to 7333584684997156864. Current Snapshot: MvccSnapshot[applied={T|T < 7333584685100064768}]
I20260926 12:32:28.521670 26721 mvcc.cc:205] Tried to move back new op lower bound from 7333584685100064768 to 7333584684997156864. Current Snapshot: MvccSnapshot[applied={T|T < 7333584685100064768}]
I20260926 12:32:28.522034 26720 mvcc.cc:205] Tried to move back new op lower bound from 7333584685100064768 to 7333584684997156864. Current Snapshot: MvccSnapshot[applied={T|T < 7333584685100064768}]
I20260926 12:32:28.530473 26238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:28.530685 26238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:28.533571 26238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1dc749948ce49449791ae3be99664dc: Sending DeleteTablet for 3 replicas of tablet 071ab6fad1c14fa68edb9a5cbca6de16
I20260926 12:32:28.534421 26492 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:28 UTC) from {username='slave'} at 127.0.0.1:46298
I20260926 12:32:28.534557 26623 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:28 UTC) from {username='slave'} at 127.0.0.1:34856
I20260926 12:32:28.534580 26361 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ab6fad1c14fa68edb9a5cbca6de16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:28 UTC) from {username='slave'} at 127.0.0.1:46026
I20260926 12:32:28.534977 26730 tablet_replica.cc:330] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: stopping tablet replica
I20260926 12:32:28.535290 26730 raft_consensus.cc:2229] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:28.535465 26729 tablet_replica.cc:330] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: stopping tablet replica
I20260926 12:32:28.535727 26730 raft_consensus.cc:2258] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:28.535751 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26299
I20260926 12:32:28.537470 26729 raft_consensus.cc:2229] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:28.537838 26729 raft_consensus.cc:2258] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:28.537957 26730 ts_tablet_manager.cc:1905] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:28.539350 26729 ts_tablet_manager.cc:1905] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:28.541034 26730 ts_tablet_manager.cc:1918] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:28.541141 26730 log.cc:1196] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/071ab6fad1c14fa68edb9a5cbca6de16
I20260926 12:32:28.541477 26730 ts_tablet_manager.cc:1939] T 071ab6fad1c14fa68edb9a5cbca6de16 P a8f8f205573e49e582dbed3681f35660: Deleting consensus metadata
I20260926 12:32:28.542445 26729 ts_tablet_manager.cc:1918] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:28.542593 26729 log.cc:1196] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/071ab6fad1c14fa68edb9a5cbca6de16
I20260926 12:32:28.542955 26729 ts_tablet_manager.cc:1939] T 071ab6fad1c14fa68edb9a5cbca6de16 P 628a26b48977456990b5715ade47a456: Deleting consensus metadata
I20260926 12:32:28.542918 26224 catalog_manager.cc:4841] TS a8f8f205573e49e582dbed3681f35660 (127.7.99.3:33513): tablet 071ab6fad1c14fa68edb9a5cbca6de16 (table test-table [id=42053cc5785f45edab6fdc29d6c3518a]) successfully deleted
I20260926 12:32:28.544032 26226 catalog_manager.cc:4841] TS 628a26b48977456990b5715ade47a456 (127.7.99.2:40505): tablet 071ab6fad1c14fa68edb9a5cbca6de16 (table test-table [id=42053cc5785f45edab6fdc29d6c3518a]) successfully deleted
W20260926 12:32:28.544472 26225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:36747 (error 108)
I20260926 12:32:28.545262 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26430
I20260926 12:32:28.545480 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:28.545156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:36747 (local address 127.0.0.1:46042)
0926 12:32:28.545285 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:28.545382 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:32:28.546783 26225 catalog_manager.cc:4584] TS 782bbbb3668d4ac6955d3c2bbf590351 (127.7.99.1:36747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 071ab6fad1c14fa68edb9a5cbca6de16: Network error: Client connection negotiation failed: client connection to 127.7.99.1:36747: connect: Connection refused (error 111)
I20260926 12:32:28.552748 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26561
I20260926 12:32:28.558498 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26208
2026-09-26T12:32:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:28 dist-test-slave-3jw2 krb5kdc[26735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:28 dist-test-slave-3jw2 krb5kdc[26735](info): set up 2 sockets
Sep 26 12:32:28 dist-test-slave-3jw2 krb5kdc[26735](info): commencing operation
krb5kdc: starting...
W20260926 12:32:29.600397 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:29 dist-test-slave-3jw2 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:29.718276 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:39671
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:29.827487 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:29.827797 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:29.827879 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:29.831869 26748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:29.831957 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:29.831996 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:29.832034 26748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:29.832083 26748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:29.837338 26748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:29.838379 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:29.839491 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:32:29.848063 26748 server_base.cc:1034] running on GCE node
W20260926 12:32:29.848016 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:29.848016 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:29.848044 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:29.848846 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:29.849931 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:29.851111 26748 hybrid_clock.cc:648] HybridClock initialized: now 1790425949851088 us; error 41 us; skew 500 ppm
Sep 26 12:32:29 dist-test-slave-3jw2 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:29.853936 26748 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:29.854771 26748 webserver.cc:460] Webserver started at http://127.7.99.62:41251/ using document root <none> and password file <none>
I20260926 12:32:29.855317 26748 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:29.855405 26748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:29.855628 26748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:29.857661 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "576455d6b3454a05938d0bfcc2c57aee"
format_stamp: "Formatted at 2026-09-26 12:32:29 on dist-test-slave-3jw2"
server_key: "1870d739dfb4c064afd01c9be9ef9502"
server_key_iv: "0f6d7a6e87066d698923ceb3a7c2581f"
server_key_version: "encryptionkey@0"
I20260926 12:32:29.858237 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "576455d6b3454a05938d0bfcc2c57aee"
format_stamp: "Formatted at 2026-09-26 12:32:29 on dist-test-slave-3jw2"
server_key: "1870d739dfb4c064afd01c9be9ef9502"
server_key_iv: "0f6d7a6e87066d698923ceb3a7c2581f"
server_key_version: "encryptionkey@0"
I20260926 12:32:29.861866 26748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:29.864313 26763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:29.865489 26748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:29.865622 26748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "576455d6b3454a05938d0bfcc2c57aee"
format_stamp: "Formatted at 2026-09-26 12:32:29 on dist-test-slave-3jw2"
server_key: "1870d739dfb4c064afd01c9be9ef9502"
server_key_iv: "0f6d7a6e87066d698923ceb3a7c2581f"
server_key_version: "encryptionkey@0"
I20260926 12:32:29.865733 26748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:29.882561 26748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:29.885560 26748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:29.885780 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:29.892756 26748 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:39671
I20260926 12:32:29.892769 26815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:39671 every 8 connection(s)
I20260926 12:32:29.893761 26748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:29.896443 26816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:29.901845 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: Bootstrap starting.
I20260926 12:32:29.903667 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26748
I20260926 12:32:29.903777 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:29.904093 7564 external_mini_cluster.cc:1428] Setting key 325afd13f59eea4e85fa36b1c3c5bf28
I20260926 12:32:29.904420 26816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:29.905259 26816 log.cc:824] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:29.907199 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: No bootstrap required, opened a new log
I20260926 12:32:29.910651 26816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } }
I20260926 12:32:29.910919 26816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:29.910984 26816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576455d6b3454a05938d0bfcc2c57aee, State: Initialized, Role: FOLLOWER
I20260926 12:32:29.911471 26816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } }
I20260926 12:32:29.911597 26816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:29.911691 26816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:29.911806 26816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 0 FOLLOWER]: Advancing to term 1
Sep 26 12:32:29 dist-test-slave-3jw2 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:29.913014 26816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } }
I20260926 12:32:29.913411 26816 leader_election.cc:302] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 576455d6b3454a05938d0bfcc2c57aee; no voters:
I20260926 12:32:29.913735 26816 leader_election.cc:288] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:29.913916 26821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:29.914170 26821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [term 1 LEADER]: Becoming Leader. State: Replica: 576455d6b3454a05938d0bfcc2c57aee, State: Running, Role: LEADER
I20260926 12:32:29.914644 26821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } }
I20260926 12:32:29.915041 26816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:29.916301 26822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "576455d6b3454a05938d0bfcc2c57aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } } }
I20260926 12:32:29.916419 26822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:29.916680 26823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 576455d6b3454a05938d0bfcc2c57aee. Latest consensus state: current_term: 1 leader_uuid: "576455d6b3454a05938d0bfcc2c57aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576455d6b3454a05938d0bfcc2c57aee" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 39671 } } }
I20260926 12:32:29.916790 26823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:29.917028 26830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:29.918931 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:29.905228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.7.99.62:39671)
0926 12:32:29.905709 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:29.905728 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:29.905781 (+ 53us) server_negotiation.cc:407] Connection header received
0926 12:32:29.906472 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:29.906489 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:29.906812 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:29.907146 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:29.907927 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:29.909001 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:29.909867 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:29.910336 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:29.910506 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:29.913469 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:29.913473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:29.913487 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:29.913538 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:29.915594 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:29.916142 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:29.916149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:29.916154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:29.916331 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:29.916793 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:29.916800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:29.916805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:29.917350 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:29.917556 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:29.917953 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 12:32:29.918252 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260926 12:32:29.920738 26830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:29.925850 26830 catalog_manager.cc:1347] Generated new cluster ID: 235c7dc2c46345b2a502e4b87ab7bb17
I20260926 12:32:29.925920 26830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:29.938670 26830 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:29.939634 26830 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:29.946949 26830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: Generated new TSK 0
I20260926 12:32:29.947620 26830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
Entry for principal kudu/127.7.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab.
I20260926 12:32:29.982951 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:30.092705 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:30.092941 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:30.092981 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:30.096802 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:30.096845 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:30.096897 26842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:30.101820 26842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.1.keytab
--principal=kudu/127.7.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:30.102727 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:30.103672 26842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:30.112967 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:30.112985 26847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:30.112984 26848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:30.113190 26842 server_base.cc:1034] running on GCE node
I20260926 12:32:30.113579 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:30.114245 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:30.115407 26842 hybrid_clock.cc:648] HybridClock initialized: now 1790425950115375 us; error 44 us; skew 500 ppm
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:30.118352 26842 init.cc:377] Logged in from keytab as kudu/127.7.99.1@KRBTEST.COM (short username kudu)
I20260926 12:32:30.119240 26842 webserver.cc:460] Webserver started at http://127.7.99.1:46081/ using document root <none> and password file <none>
I20260926 12:32:30.119827 26842 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:30.119935 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:30.120232 26842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:30.122229 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "fca796bdea0e4e77a5f673fb314399c9"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "ee19c38e3835d00463ab68f7300a4687"
server_key_iv: "5adb42d6f754e7096c193843c655d238"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.122769 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "fca796bdea0e4e77a5f673fb314399c9"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "ee19c38e3835d00463ab68f7300a4687"
server_key_iv: "5adb42d6f754e7096c193843c655d238"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.126403 26842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:32:30.128929 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.130270 26842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:32:30.130416 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "fca796bdea0e4e77a5f673fb314399c9"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "ee19c38e3835d00463ab68f7300a4687"
server_key_iv: "5adb42d6f754e7096c193843c655d238"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.130530 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:30.144341 26842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:30.147004 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:30.147193 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:30.147743 26842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:30.148599 26842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:30.148671 26842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.148741 26842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:30.148794 26842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.157486 26842 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:38813
I20260926 12:32:30.157508 26970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:38813 every 8 connection(s)
I20260926 12:32:30.158317 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:30.159375 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26842
I20260926 12:32:30.159511 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:30.159835 7564 external_mini_cluster.cc:1428] Setting key c433e9a4121ffa2e498142dd1a206cad
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.1@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.2@KRBTEST.COM" created.
I20260926 12:32:30.173334 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.160295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:48691 (local address 127.7.99.62:39671)
0926 12:32:30.160458 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.160462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.161290 (+ 828us) server_negotiation.cc:407] Connection header received
0926 12:32:30.162617 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.162620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.162670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.162746 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:30.164128 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.165060 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.166231 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.166669 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.166799 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.169536 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:30.169543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:30.169547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:30.169577 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:30.170982 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.171535 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.171538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.171539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.171614 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.172060 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.172063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.172064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.172319 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:30.172418 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.172920 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.173069 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:32:30.173956 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.160701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:39671 (local address 127.7.99.1:48691)
0926 12:32:30.161139 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.161203 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.162416 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.162917 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.162924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.163379 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:30.163970 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.163987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.165208 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.165211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.166104 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.166113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.166823 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.166825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.166956 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.167529 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:30.167545 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:30.169351 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:30.171089 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.171094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.171129 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.171445 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.171695 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.171697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.171699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.171921 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.172438 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:30.172443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:30.172680 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.172839 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.173085 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":136,"threads_started":1}
I20260926 12:32:30.175278 26971 heartbeater.cc:347] Connected to a master server at 127.7.99.62:39671
I20260926 12:32:30.175529 26971 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:30.176111 26971 heartbeater.cc:511] Master 127.7.99.62:39671 requested a full tablet report, sending...
I20260926 12:32:30.177886 26780 ts_manager.cc:194] Registered new tserver with Master: fca796bdea0e4e77a5f673fb314399c9 (127.7.99.1:38813)
I20260926 12:32:30.179070 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.1@KRBTEST.COM'} at 127.7.99.1:48691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
Entry for principal kudu/127.7.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab.
I20260926 12:32:30.187150 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:30.294623 26976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:30.294870 26976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:30.294911 26976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:30.298904 26976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:30.298954 26976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:30.299041 26976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:30.304117 26976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.2.keytab
--principal=kudu/127.7.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:30.305075 26976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:30.306041 26976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:30.316108 26984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:30.316114 26982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:30.316419 26976 server_base.cc:1034] running on GCE node
W20260926 12:32:30.316533 26981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:30.316903 26976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:30.317543 26976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:30.318703 26976 hybrid_clock.cc:648] HybridClock initialized: now 1790425950318670 us; error 49 us; skew 500 ppm
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:30.321460 26976 init.cc:377] Logged in from keytab as kudu/127.7.99.2@KRBTEST.COM (short username kudu)
I20260926 12:32:30.322305 26976 webserver.cc:460] Webserver started at http://127.7.99.2:38387/ using document root <none> and password file <none>
I20260926 12:32:30.322840 26976 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:30.322929 26976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:30.323154 26976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:30.325173 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "cc5ac4c7f02844408894c90425e4efbe"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "74fe5aa9f84e6fc4a651b1df04ff4497"
server_key_iv: "e1323dd6df5090086147acfa9150fb52"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.325724 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "cc5ac4c7f02844408894c90425e4efbe"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "74fe5aa9f84e6fc4a651b1df04ff4497"
server_key_iv: "e1323dd6df5090086147acfa9150fb52"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.329396 26976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:32:30.331893 26991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.333117 26976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:32:30.333261 26976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "cc5ac4c7f02844408894c90425e4efbe"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "74fe5aa9f84e6fc4a651b1df04ff4497"
server_key_iv: "e1323dd6df5090086147acfa9150fb52"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.333369 26976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:30.370891 26976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:30.374105 26976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:30.374308 26976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:30.374943 26976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:30.375869 26976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:30.375937 26976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.376001 26976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:30.376031 26976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.385092 26976 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:38425
I20260926 12:32:30.385211 27104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:38425 every 8 connection(s)
I20260926 12:32:30.385979 26976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:30.393630 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 26976
I20260926 12:32:30.393738 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:30.394009 7564 external_mini_cluster.cc:1428] Setting key 5ed47083d26445ee8c7b9bf52ed56ebd
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.2@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:30.400065 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.387773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:55603 (local address 127.7.99.62:39671)
0926 12:32:30.387923 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.387928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.388822 (+ 894us) server_negotiation.cc:407] Connection header received
0926 12:32:30.390105 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.390108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.390153 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.390226 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:30.391374 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.391900 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.393079 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.393300 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.393403 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.396136 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:30.396141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:30.396147 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:30.396201 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:30.397937 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.398433 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.398440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.398444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.398535 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.398839 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.398846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.398850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.399102 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:30.399213 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.399674 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.399816 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:32:30.400820 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.388154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:39671 (local address 127.7.99.2:55603)
0926 12:32:30.388666 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.388707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.389915 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.390408 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.390414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.390786 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:30.391248 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.391263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.392044 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.392047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.392965 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.392975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.393421 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.393424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.393564 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.394232 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:30.394248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:30.395978 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:30.398066 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.398071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.398083 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.398340 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.398622 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.398625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.398626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.398749 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.399210 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:30.399215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:30.399439 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.399590 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.399913 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260926 12:32:30.401943 27105 heartbeater.cc:347] Connected to a master server at 127.7.99.62:39671
I20260926 12:32:30.402202 27105 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:30.402779 27105 heartbeater.cc:511] Master 127.7.99.62:39671 requested a full tablet report, sending...
I20260926 12:32:30.403893 26780 ts_manager.cc:194] Registered new tserver with Master: cc5ac4c7f02844408894c90425e4efbe (127.7.99.2:38425)
WARNING: no policy specified for kudu/127.7.99.3@KRBTEST.COM; defaulting to no policy
I20260926 12:32:30.404544 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.2@KRBTEST.COM'} at 127.7.99.2:55603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
Entry for principal kudu/127.7.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab.
I20260926 12:32:30.422523 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:30.532020 27110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:30.532331 27110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:30.532404 27110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:30.536724 27110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:30.536811 27110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:30.536900 27110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:30.542176 27110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.3.keytab
--principal=kudu/127.7.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:30.543272 27110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:30.544330 27110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:30.553791 27115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:30.553784 27118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:30.553784 27116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:30.554204 27110 server_base.cc:1034] running on GCE node
I20260926 12:32:30.554684 27110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:30.555289 27110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:30.556447 27110 hybrid_clock.cc:648] HybridClock initialized: now 1790425950556432 us; error 26 us; skew 500 ppm
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:30.559293 27110 init.cc:377] Logged in from keytab as kudu/127.7.99.3@KRBTEST.COM (short username kudu)
I20260926 12:32:30.560129 27110 webserver.cc:460] Webserver started at http://127.7.99.3:44039/ using document root <none> and password file <none>
I20260926 12:32:30.560717 27110 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:30.560787 27110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:30.560952 27110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:30.562903 27110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "83704013c03244a289aa584342df5da1"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "b492fe3f9cc11323207cac4292409194"
server_key_iv: "9896cb86d377a164b36b7775e89feca2"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.563401 27110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "83704013c03244a289aa584342df5da1"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "b492fe3f9cc11323207cac4292409194"
server_key_iv: "9896cb86d377a164b36b7775e89feca2"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.567010 27110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:32:30.569504 27125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.570765 27110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:30.570935 27110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "83704013c03244a289aa584342df5da1"
format_stamp: "Formatted at 2026-09-26 12:32:30 on dist-test-slave-3jw2"
server_key: "b492fe3f9cc11323207cac4292409194"
server_key_iv: "9896cb86d377a164b36b7775e89feca2"
server_key_version: "encryptionkey@0"
I20260926 12:32:30.571045 27110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:30.583146 27110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:30.586504 27110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:30.586715 27110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:30.587319 27110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:30.588281 27110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:30.588347 27110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.588415 27110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:30.588497 27110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:30.597169 27110 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:39245
I20260926 12:32:30.597224 27238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:39245 every 8 connection(s)
I20260926 12:32:30.598129 27110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
Sep 26 12:32:30 dist-test-slave-3jw2 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425950, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.3@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:30.608275 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27110
I20260926 12:32:30.608419 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:30.608770 7564 external_mini_cluster.cc:1428] Setting key 9eb8d415b6eb39090a568668b86abbbe
I20260926 12:32:30.612020 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.600051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:59147 (local address 127.7.99.62:39671)
0926 12:32:30.600302 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.600308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.601107 (+ 799us) server_negotiation.cc:407] Connection header received
0926 12:32:30.602505 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.602517 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.602576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.602662 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:30.603841 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.604455 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.605648 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.605877 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.605971 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.608559 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:30.608562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:30.608565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:30.608590 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:30.610088 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.610601 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.610607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.610619 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.610729 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.611103 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.611105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.611106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.611274 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:30.611346 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.611814 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.611906 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 12:32:30.612861 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.600460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:39671 (local address 127.7.99.3:59147)
0926 12:32:30.600930 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.600974 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.602275 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.602823 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.602829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.603213 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:30.603688 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.603701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.604612 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.604616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.605489 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.605498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.606016 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.606019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.606187 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.606737 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:30.606753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:30.608358 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:30.610220 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.610226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.610240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.610489 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.610844 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:30.610847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:30.610848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:30.611017 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:30.611359 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:30.611364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:30.611615 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.611769 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.612009 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260926 12:32:30.614065 27239 heartbeater.cc:347] Connected to a master server at 127.7.99.62:39671
I20260926 12:32:30.614431 27239 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:30.614939 27239 heartbeater.cc:511] Master 127.7.99.62:39671 requested a full tablet report, sending...
I20260926 12:32:30.616034 26780 ts_manager.cc:194] Registered new tserver with Master: 83704013c03244a289aa584342df5da1 (127.7.99.3:39245)
I20260926 12:32:30.616798 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.99.3@KRBTEST.COM'} at 127.7.99.3:59147
I20260926 12:32:30.622391 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:30.631878 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.624328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.7.99.62:39671)
0926 12:32:30.624440 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.624444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.624691 (+ 247us) server_negotiation.cc:407] Connection header received
0926 12:32:30.624849 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.624852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.624899 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.624975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:30.625661 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.626168 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.627270 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.627518 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.627634 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.628986 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:30.628990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:30.628993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:30.629018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:30.630235 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.630655 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.630658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.630660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.630743 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:30.631016 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:30.631020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:30.631022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:30.631233 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:30.631369 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.631598 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.631704 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:32:30.634785 26780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:30.638119 26780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:30.652014 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.646025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:39245 (local address 127.0.0.1:37710)
0926 12:32:30.646506 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.646553 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.646682 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.647716 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.647722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.647744 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:30.647894 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.647902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649179 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.649186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.650518 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.650527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.651558 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.651561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.651652 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.651681 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.651768 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.651856 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":81,"thread_start_us":63,"threads_started":1}
I20260926 12:32:30.652065 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.646181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.7.99.3:39245)
0926 12:32:30.647238 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.647250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.647273 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:32:30.647344 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.647349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.647545 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.648036 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:30.648268 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649045 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.650670 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.651433 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.651544 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.651697 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.651799 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.651894 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":139,"threads_started":1}
I20260926 12:32:30.652704 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.646693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38813 (local address 127.0.0.1:32956)
0926 12:32:30.646943 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.646957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.647046 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.647825 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.647829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.647847 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:30.647995 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.648002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649669 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.649673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.650943 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.650954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.652013 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.652016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.652111 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.652505 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.652550 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.652612 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":47,"thread_start_us":96,"threads_started":1}
I20260926 12:32:30.653604 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.646195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:38425 (local address 127.0.0.1:34004)
0926 12:32:30.646901 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.646915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.646996 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.647839 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.647841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.647853 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:30.647995 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.648002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649522 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.649526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.650947 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.650954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.653211 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.653215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.653317 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.653339 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.653411 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.653474 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":184,"thread_start_us":119,"threads_started":1}
I20260926 12:32:30.653764 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.647120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.7.99.1:38813)
0926 12:32:30.647419 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.647431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.647451 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:32:30.647518 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.647524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.647679 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.648136 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:30.648341 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649425 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.651071 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.651888 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.651997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.653439 (+ 1442us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.653491 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.653593 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
I20260926 12:32:30.654765 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.646759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.7.99.2:38425)
0926 12:32:30.647186 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.647195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.647215 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:32:30.647300 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.647306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.647514 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.647660 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:30.648122 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.649367 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.652306 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.653068 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.653193 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.653306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.654511 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.654624 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":141,"threads_started":1}
I20260926 12:32:30.655470 27173 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 (DEFAULT_TABLE table=test-table [id=9d5e84a2d41b49a694b0733e2ab13b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:30.656692 27173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:30.656395 26905 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 (DEFAULT_TABLE table=test-table [id=9d5e84a2d41b49a694b0733e2ab13b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:30.657430 26905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:30.657379 27039 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 (DEFAULT_TABLE table=test-table [id=9d5e84a2d41b49a694b0733e2ab13b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:30.658280 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:30.663280 27256 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: Bootstrap starting.
I20260926 12:32:30.664588 27257 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Bootstrap starting.
I20260926 12:32:30.664746 27258 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Bootstrap starting.
I20260926 12:32:30.665728 27256 tablet_bootstrap.cc:654] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:30.666419 27257 tablet_bootstrap.cc:654] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:30.666818 27258 tablet_bootstrap.cc:654] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:30.666983 27256 log.cc:824] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:30.667515 27257 log.cc:824] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:30.668028 27258 log.cc:824] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:30.669428 27256 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: No bootstrap required, opened a new log
I20260926 12:32:30.669632 27256 ts_tablet_manager.cc:1397] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 12:32:30.669973 27257 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: No bootstrap required, opened a new log
I20260926 12:32:30.670003 27258 tablet_bootstrap.cc:492] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: No bootstrap required, opened a new log
I20260926 12:32:30.670172 27257 ts_tablet_manager.cc:1397] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:32:30.670204 27258 ts_tablet_manager.cc:1397] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:32:30.673017 27258 raft_consensus.cc:348] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.673267 27258 raft_consensus.cc:374] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:30.673328 27258 raft_consensus.cc:729] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ac4c7f02844408894c90425e4efbe, State: Initialized, Role: FOLLOWER
I20260926 12:32:30.673544 27256 raft_consensus.cc:348] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.673796 27256 raft_consensus.cc:374] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:30.673859 27256 raft_consensus.cc:729] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca796bdea0e4e77a5f673fb314399c9, State: Initialized, Role: FOLLOWER
I20260926 12:32:30.673807 27258 consensus_queue.cc:260] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.673821 27257 raft_consensus.cc:348] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.674059 27257 raft_consensus.cc:374] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:30.674114 27257 raft_consensus.cc:729] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83704013c03244a289aa584342df5da1, State: Initialized, Role: FOLLOWER
I20260926 12:32:30.674268 27256 consensus_queue.cc:260] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.674474 27257 consensus_queue.cc:260] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.675118 27258 ts_tablet_manager.cc:1428] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:32:30.675251 27257 ts_tablet_manager.cc:1428] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:32:30.675251 27256 ts_tablet_manager.cc:1428] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:32:30.675499 27105 heartbeater.cc:503] Master 127.7.99.62:39671 was elected leader, sending a full tablet report...
I20260926 12:32:30.675665 27239 heartbeater.cc:503] Master 127.7.99.62:39671 was elected leader, sending a full tablet report...
I20260926 12:32:30.675676 26971 heartbeater.cc:503] Master 127.7.99.62:39671 was elected leader, sending a full tablet report...
I20260926 12:32:30.808440 27262 raft_consensus.cc:482] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:30.808665 27262 raft_consensus.cc:504] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.809844 27262 leader_election.cc:288] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83704013c03244a289aa584342df5da1 (127.7.99.3:39245), fca796bdea0e4e77a5f673fb314399c9 (127.7.99.1:38813)
I20260926 12:32:30.813936 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.810248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:36367 (local address 127.7.99.3:39245)
0926 12:32:30.810393 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.810399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.810415 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:32:30.810588 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.810590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.810649 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.810716 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:30.811179 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.811799 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.812894 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.813360 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.813439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.813526 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.813732 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.813805 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:32:30.813941 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.810164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:39245 (local address 127.7.99.2:36367)
0926 12:32:30.810327 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.810345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.810453 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.810842 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.810844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.810885 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:30.811017 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.811023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.811912 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.811915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.812777 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.812785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.813478 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.813484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.813586 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.813614 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.813684 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.813773 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3,"spinlock_wait_cycles":2432}
I20260926 12:32:30.813961 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.810576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:33377 (local address 127.7.99.1:38813)
0926 12:32:30.810716 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.810720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.810893 (+ 173us) server_negotiation.cc:407] Connection header received
0926 12:32:30.810990 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.810993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.811039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.811116 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:30.811417 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.811927 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.813016 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.813394 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.813520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.813585 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.813778 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.813857 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:32:30.813961 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.810446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38813 (local address 127.7.99.2:33377)
0926 12:32:30.810797 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.810809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:30.810908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:30.811175 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:30.811177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:30.811195 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:30.811308 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.811316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.812035 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.812038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.812898 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:30.812908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.813549 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.813553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:30.813635 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.813652 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:32:30.813742 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:32:30.813804 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"spinlock_wait_cycles":4736,"thread_start_us":141,"threads_started":1}
I20260926 12:32:30.814862 27193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0b6feb2c4f4a4faf9b8d34a0033c55" candidate_uuid: "cc5ac4c7f02844408894c90425e4efbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83704013c03244a289aa584342df5da1" is_pre_election: true
I20260926 12:32:30.814864 26925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0b6feb2c4f4a4faf9b8d34a0033c55" candidate_uuid: "cc5ac4c7f02844408894c90425e4efbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca796bdea0e4e77a5f673fb314399c9" is_pre_election: true
I20260926 12:32:30.815137 26925 raft_consensus.cc:2454] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc5ac4c7f02844408894c90425e4efbe in term 0.
I20260926 12:32:30.815147 27193 raft_consensus.cc:2454] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc5ac4c7f02844408894c90425e4efbe in term 0.
I20260926 12:32:30.815615 26994 leader_election.cc:302] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83704013c03244a289aa584342df5da1, cc5ac4c7f02844408894c90425e4efbe; no voters:
I20260926 12:32:30.815912 27262 raft_consensus.cc:2781] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:30.815982 27262 raft_consensus.cc:482] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:30.816030 27262 raft_consensus.cc:3037] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:30.817147 27262 raft_consensus.cc:504] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.817480 27262 leader_election.cc:288] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [CANDIDATE]: Term 1 election: Requested vote from peers 83704013c03244a289aa584342df5da1 (127.7.99.3:39245), fca796bdea0e4e77a5f673fb314399c9 (127.7.99.1:38813)
I20260926 12:32:30.817873 27193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0b6feb2c4f4a4faf9b8d34a0033c55" candidate_uuid: "cc5ac4c7f02844408894c90425e4efbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83704013c03244a289aa584342df5da1"
I20260926 12:32:30.817979 27193 raft_consensus.cc:3037] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:30.818009 26925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0b6feb2c4f4a4faf9b8d34a0033c55" candidate_uuid: "cc5ac4c7f02844408894c90425e4efbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca796bdea0e4e77a5f673fb314399c9"
I20260926 12:32:30.818188 26925 raft_consensus.cc:3037] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:30.819032 27193 raft_consensus.cc:2454] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5ac4c7f02844408894c90425e4efbe in term 1.
I20260926 12:32:30.819228 26925 raft_consensus.cc:2454] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5ac4c7f02844408894c90425e4efbe in term 1.
I20260926 12:32:30.819335 26994 leader_election.cc:302] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83704013c03244a289aa584342df5da1, cc5ac4c7f02844408894c90425e4efbe; no voters:
I20260926 12:32:30.819499 27262 raft_consensus.cc:2781] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:30.819751 27262 raft_consensus.cc:686] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 1 LEADER]: Becoming Leader. State: Replica: cc5ac4c7f02844408894c90425e4efbe, State: Running, Role: LEADER
I20260926 12:32:30.820091 27262 consensus_queue.cc:237] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } }
I20260926 12:32:30.823192 26777 catalog_manager.cc:5495] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe reported cstate change: term changed from 0 to 1, leader changed from <none> to cc5ac4c7f02844408894c90425e4efbe (127.7.99.2). New cstate: current_term: 1 leader_uuid: "cc5ac4c7f02844408894c90425e4efbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc5ac4c7f02844408894c90425e4efbe" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 } health_report { overall_health: UNKNOWN } } }
I20260926 12:32:30.839725 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.836269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.7.99.2:38425)
0926 12:32:30.836371 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:30.836377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:30.836391 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:32:30.836523 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:30.836525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:30.836573 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:30.836668 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:30.836958 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.837568 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.838464 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:30.838681 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:30.838763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:30.839011 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:30.839422 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:30.839521 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:30.839563 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 12:32:30.839622 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 12:32:30.845341 27193 raft_consensus.cc:1261] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 1 FOLLOWER]: Refusing update from remote peer cc5ac4c7f02844408894c90425e4efbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:30.845359 26925 raft_consensus.cc:1261] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9 [term 1 FOLLOWER]: Refusing update from remote peer cc5ac4c7f02844408894c90425e4efbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:30.846048 27262 consensus_queue.cc:1035] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca796bdea0e4e77a5f673fb314399c9" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 38813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:30.846283 27266 consensus_queue.cc:1035] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "83704013c03244a289aa584342df5da1" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:32:30.849010 27240 tablet.cc:2369] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:30.854373 27270 mvcc.cc:205] Tried to move back new op lower bound from 7333584694655590400 to 7333584694561701888. Current Snapshot: MvccSnapshot[applied={T|T < 7333584694655590400}]
I20260926 12:32:30.855494 27271 mvcc.cc:205] Tried to move back new op lower bound from 7333584694655590400 to 7333584694561701888. Current Snapshot: MvccSnapshot[applied={T|T < 7333584694655590400}]
I20260926 12:32:30.862772 26777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:30.862949 26777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:30.865629 26777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 576455d6b3454a05938d0bfcc2c57aee: Sending DeleteTablet for 3 replicas of tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55
I20260926 12:32:30.866496 27039 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 12:32:30.866636 27173 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:37710
I20260926 12:32:30.866636 26905 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:30 UTC) from {username='kudu', principal='kudu/127.7.99.62@KRBTEST.COM'} at 127.0.0.1:32956
I20260926 12:32:30.866999 27278 tablet_replica.cc:330] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: stopping tablet replica
I20260926 12:32:30.867297 27278 raft_consensus.cc:2229] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:30.867383 27280 tablet_replica.cc:330] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P fca796bdea0e4e77a5f673fb314399c9: stopping tablet replica
I20260926 12:32:30.867501 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26842
I20260926 12:32:30.869951 27278 raft_consensus.cc:2258] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:30.870458 27279 tablet_replica.cc:330] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: stopping tablet replica
I20260926 12:32:30.870764 27279 raft_consensus.cc:2229] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:30.871054 27279 raft_consensus.cc:2258] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:30.872545 27279 ts_tablet_manager.cc:1905] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:30.873589 27278 ts_tablet_manager.cc:1905] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P cc5ac4c7f02844408894c90425e4efbe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:32:30.875087 26766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:38813 (error 108)
I20260926 12:32:30.875336 27279 ts_tablet_manager.cc:1918] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:30.875437 27279 log.cc:1196] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/5e0b6feb2c4f4a4faf9b8d34a0033c55
I20260926 12:32:30.875790 27279 ts_tablet_manager.cc:1939] T 5e0b6feb2c4f4a4faf9b8d34a0033c55 P 83704013c03244a289aa584342df5da1: Deleting consensus metadata
I20260926 12:32:30.875929 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26976
I20260926 12:32:30.877621 26766 catalog_manager.cc:4841] TS 83704013c03244a289aa584342df5da1 (127.7.99.3:39245): tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55 (table test-table [id=9d5e84a2d41b49a694b0733e2ab13b21]) successfully deleted
I20260926 12:32:30.877794 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.877497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:38813 (local address 127.0.0.1:32960)
0926 12:32:30.877638 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.877711 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 12:32:30.877943 26766 catalog_manager.cc:4584] TS fca796bdea0e4e77a5f673fb314399c9 (127.7.99.1:38813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e0b6feb2c4f4a4faf9b8d34a0033c55: Network error: Client connection negotiation failed: client connection to 127.7.99.1:38813: connect: Connection refused (error 111)
I20260926 12:32:30.881253 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27110
I20260926 12:32:30.881979 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:30.881377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:38425 (local address 127.0.0.1:34026)
0926 12:32:30.881868 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:30.881915 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.2:38425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260926 12:32:30.888540 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 26748
2026-09-26T12:32:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2331 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T12:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:30Z Disabled control of system clock
I20260926 12:32:30.920152 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:35787
--webserver_interface=127.7.99.62
--webserver_port=0
--builtin_ntp_servers=127.7.99.20:40631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:31.030900 27285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:31.031157 27285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:31.031193 27285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:31.035131 27285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:31.035192 27285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:31.035209 27285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:31.035228 27285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:31.035247 27285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:31.039827 27285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:35787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:31.040776 27285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:31.042062 27285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:31.050336 27291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.050354 27293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.050336 27290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:31.050594 27285 server_base.cc:1034] running on GCE node
I20260926 12:32:31.051234 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:31.052295 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:31.053473 27285 hybrid_clock.cc:648] HybridClock initialized: now 1790425951053459 us; error 28 us; skew 500 ppm
I20260926 12:32:31.055161 27285 webserver.cc:460] Webserver started at http://127.7.99.62:37025/ using document root <none> and password file <none>
I20260926 12:32:31.055702 27285 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:31.055794 27285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:31.056015 27285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:31.057982 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "e127faa852c19dfda1936c4a51724dab"
server_key_iv: "89e029d9cda5aaac058cd0ca81e07136"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.058522 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "e127faa852c19dfda1936c4a51724dab"
server_key_iv: "89e029d9cda5aaac058cd0ca81e07136"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.061995 27285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:31.064321 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.065464 27285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:31.065599 27285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "e127faa852c19dfda1936c4a51724dab"
server_key_iv: "89e029d9cda5aaac058cd0ca81e07136"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.065740 27285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:31.087946 27285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:31.088668 27285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:31.088863 27285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:31.095556 27285 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:35787
I20260926 12:32:31.095564 27351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:35787 every 8 connection(s)
I20260926 12:32:31.096554 27285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:31.099517 27352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:31.104801 27352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: Bootstrap starting.
I20260926 12:32:31.106021 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27285
I20260926 12:32:31.106149 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:31.106436 7564 external_mini_cluster.cc:1428] Setting key cb0dd08278ebb7d78bb946607b586781
I20260926 12:32:31.107366 27352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:31.108281 27352 log.cc:824] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:31.110361 27352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: No bootstrap required, opened a new log
I20260926 12:32:31.114473 27352 raft_consensus.cc:348] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } }
I20260926 12:32:31.114727 27352 raft_consensus.cc:374] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:31.114797 27352 raft_consensus.cc:729] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54f18b5d6d94745b8c5ca7f12a8ef19, State: Initialized, Role: FOLLOWER
I20260926 12:32:31.115289 27352 consensus_queue.cc:260] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } }
I20260926 12:32:31.115443 27352 raft_consensus.cc:388] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:31.115499 27352 raft_consensus.cc:482] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:31.115588 27352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:31.116416 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.107496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.7.99.62:35787)
0926 12:32:31.108153 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.108191 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.108232 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:32:31.108960 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.109004 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.109393 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.109738 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:31.110155 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.111575 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.112478 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.113179 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.113377 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.114091 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:31.114097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:31.114112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:31.114572 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:31.114650 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:31.114677 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:31.114778 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.115129 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.115378 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260926 12:32:31.116914 27352 raft_consensus.cc:504] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } }
I20260926 12:32:31.117268 27352 leader_election.cc:302] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f54f18b5d6d94745b8c5ca7f12a8ef19; no voters:
I20260926 12:32:31.117574 27352 leader_election.cc:288] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:31.117650 27357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:31.117835 27357 raft_consensus.cc:686] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [term 1 LEADER]: Becoming Leader. State: Replica: f54f18b5d6d94745b8c5ca7f12a8ef19, State: Running, Role: LEADER
I20260926 12:32:31.118115 27357 consensus_queue.cc:237] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } }
I20260926 12:32:31.118764 27352 sys_catalog.cc:564] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:31.119678 27358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } } }
I20260926 12:32:31.119801 27358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:31.120220 27366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:31.120195 27359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f54f18b5d6d94745b8c5ca7f12a8ef19. Latest consensus state: current_term: 1 leader_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54f18b5d6d94745b8c5ca7f12a8ef19" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 35787 } } }
I20260926 12:32:31.120896 27359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:31.123085 27366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:31.128307 27366 catalog_manager.cc:1347] Generated new cluster ID: 76bbcfbce8944a02a0b56ac8a662dbaf
I20260926 12:32:31.128374 27366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:31.145913 27366 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:31.146957 27366 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:31.157372 27366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: Generated new TSK 0
I20260926 12:32:31.158252 27366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:31.169332 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.1:0
--local_ip_for_outbound_sockets=127.7.99.1
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--builtin_ntp_servers=127.7.99.20:40631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:32:31.278365 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:31.278654 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:31.278738 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:31.282720 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:31.282796 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:31.282886 27376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.1
I20260926 12:32:31.287535 27376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.99.1
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.1
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:31.288573 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:31.289916 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:31.299026 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.299046 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:31.299301 27376 server_base.cc:1034] running on GCE node
W20260926 12:32:31.299086 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:31.299685 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:31.300302 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:31.301460 27376 hybrid_clock.cc:648] HybridClock initialized: now 1790425951301441 us; error 32 us; skew 500 ppm
I20260926 12:32:31.303165 27376 webserver.cc:460] Webserver started at http://127.7.99.1:40905/ using document root <none> and password file <none>
I20260926 12:32:31.303702 27376 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:31.303762 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:31.304003 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:31.305979 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data/instance:
uuid: "2992f00b14254ed087669c7356b266d0"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "297d971940add11d9ed1ef9604951e8c"
server_key_iv: "e2ee9245f80758397042df2ef260319a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.306510 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance:
uuid: "2992f00b14254ed087669c7356b266d0"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "297d971940add11d9ed1ef9604951e8c"
server_key_iv: "e2ee9245f80758397042df2ef260319a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.309944 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:31.312235 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.313360 27376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:31.313498 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal
uuid: "2992f00b14254ed087669c7356b266d0"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "297d971940add11d9ed1ef9604951e8c"
server_key_iv: "e2ee9245f80758397042df2ef260319a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.313604 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:31.334824 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:31.335469 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:31.335655 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:31.336268 27376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:31.337145 27376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:31.337214 27376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.337291 27376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:31.337327 27376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.346634 27376 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.1:34941
I20260926 12:32:31.346743 27503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.1:34941 every 8 connection(s)
I20260926 12:32:31.347514 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/data/info.pb
I20260926 12:32:31.356001 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27376
I20260926 12:32:31.356247 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-0/wal/instance
I20260926 12:32:31.356616 7564 external_mini_cluster.cc:1428] Setting key 0357bd336a87fb37b4fbc5bc2ebf34a6
I20260926 12:32:31.357035 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.349440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.1:42237 (local address 127.7.99.62:35787)
0926 12:32:31.349608 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.349613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.350694 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 12:32:31.352013 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.352016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.352066 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.352152 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:31.353139 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.353714 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.354884 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.355108 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.355201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.356450 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:31.356455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:31.356459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:31.356475 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:31.356500 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:31.356510 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:31.356615 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.356822 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.356888 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:32:31.357872 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.349874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35787 (local address 127.7.99.1:42237)
0926 12:32:31.350530 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.350569 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.351807 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.352396 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.352403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.352487 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:31.352976 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.352989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.353853 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.353856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.354749 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.354759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.355279 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.355282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.355434 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.356094 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:31.356096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:31.356179 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:31.356286 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:31.356625 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:31.356633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:31.356638 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.356801 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.357054 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260926 12:32:31.358704 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.2:0
--local_ip_for_outbound_sockets=127.7.99.2
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--builtin_ntp_servers=127.7.99.20:40631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:31.359112 27504 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35787
I20260926 12:32:31.359347 27504 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:31.359838 27504 heartbeater.cc:511] Master 127.7.99.62:35787 requested a full tablet report, sending...
I20260926 12:32:31.361389 27316 ts_manager.cc:194] Registered new tserver with Master: 2992f00b14254ed087669c7356b266d0 (127.7.99.1:34941)
I20260926 12:32:31.362593 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.1:42237
W20260926 12:32:31.469841 27507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:31.470161 27507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:31.470239 27507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:31.474243 27507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:31.474304 27507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:31.474361 27507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.2
I20260926 12:32:31.479036 27507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.99.2
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.2
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:31.480147 27507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:31.481580 27507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:32:31.491099 27507 server_base.cc:1034] running on GCE node
W20260926 12:32:31.491047 27512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.491016 27513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.491011 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:31.491633 27507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:31.492339 27507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:31.493518 27507 hybrid_clock.cc:648] HybridClock initialized: now 1790425951493501 us; error 28 us; skew 500 ppm
I20260926 12:32:31.495316 27507 webserver.cc:460] Webserver started at http://127.7.99.2:40645/ using document root <none> and password file <none>
I20260926 12:32:31.495923 27507 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:31.495987 27507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:31.496146 27507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:31.498122 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data/instance:
uuid: "3b934ccf7d074bc788f90e12dd3e7e12"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "54f2182624bc74a2bddaca68baf8d174"
server_key_iv: "d533e2ce47e8103bdd3587c9a454ad40"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.498618 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance:
uuid: "3b934ccf7d074bc788f90e12dd3e7e12"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "54f2182624bc74a2bddaca68baf8d174"
server_key_iv: "d533e2ce47e8103bdd3587c9a454ad40"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.502090 27507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:31.504410 27521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.505641 27507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:31.505731 27507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal
uuid: "3b934ccf7d074bc788f90e12dd3e7e12"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "54f2182624bc74a2bddaca68baf8d174"
server_key_iv: "d533e2ce47e8103bdd3587c9a454ad40"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.505822 27507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:31.530169 27507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:31.530879 27507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:31.531030 27507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:31.531585 27507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:31.532505 27507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:31.532553 27507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.532594 27507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:31.532609 27507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.541662 27507 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.2:39229
I20260926 12:32:31.541690 27634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.2:39229 every 8 connection(s)
I20260926 12:32:31.542589 27507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/data/info.pb
I20260926 12:32:31.545429 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27507
I20260926 12:32:31.545571 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal/instance
I20260926 12:32:31.545882 7564 external_mini_cluster.cc:1428] Setting key 7ed8320c0e965e8897f0e04290d2fb5e
I20260926 12:32:31.548357 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.99.3:0
--local_ip_for_outbound_sockets=127.7.99.3
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--builtin_ntp_servers=127.7.99.20:40631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:32:31.553966 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.544803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:33907 (local address 127.7.99.62:35787)
0926 12:32:31.544963 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.544967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.546074 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 12:32:31.547904 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.547908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.547961 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.548053 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:31.549083 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.550060 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.551604 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.552014 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.552156 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.553359 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:31.553366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:31.553369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:31.553389 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:31.553416 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:31.553429 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:31.553515 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.553671 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.553750 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:32:31.554768 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.545160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35787 (local address 127.7.99.2:33907)
0926 12:32:31.545912 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.545966 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.547660 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.548278 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.548286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.548375 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:31.548938 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.548953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.550219 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.550225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.551475 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.551485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.552245 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.552255 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.552404 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.553077 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:31.553079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:31.553120 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:31.553190 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:31.553528 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:31.553535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:31.553539 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.553635 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.553879 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260926 12:32:31.556056 27635 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35787
I20260926 12:32:31.556346 27635 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:31.556890 27635 heartbeater.cc:511] Master 127.7.99.62:35787 requested a full tablet report, sending...
I20260926 12:32:31.557962 27316 ts_manager.cc:194] Registered new tserver with Master: 3b934ccf7d074bc788f90e12dd3e7e12 (127.7.99.2:39229)
I20260926 12:32:31.558543 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.2:33907
W20260926 12:32:31.662648 27638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:31.662921 27638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:31.662961 27638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:31.667207 27638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:31.667268 27638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:31.667323 27638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.99.3
I20260926 12:32:31.672472 27638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.99.3
--webserver_port=0
--tserver_master_addrs=127.7.99.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.99.3
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:31.673609 27638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:31.675138 27638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:31.684790 27646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.684790 27643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:31.684795 27644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:31.685047 27638 server_base.cc:1034] running on GCE node
I20260926 12:32:31.685577 27638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:31.686172 27638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:31.687330 27638 hybrid_clock.cc:648] HybridClock initialized: now 1790425951687303 us; error 39 us; skew 500 ppm
I20260926 12:32:31.689175 27638 webserver.cc:460] Webserver started at http://127.7.99.3:37765/ using document root <none> and password file <none>
I20260926 12:32:31.689754 27638 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:31.689813 27638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:31.690076 27638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:31.692134 27638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data/instance:
uuid: "ff77ab3625bb43488ef3bd605669c765"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "eb5c830bcc7b4f27e85069f3b6aa8bd2"
server_key_iv: "6199911f9e1ea92987b9c4acc0fd5a1a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.692862 27638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance:
uuid: "ff77ab3625bb43488ef3bd605669c765"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "eb5c830bcc7b4f27e85069f3b6aa8bd2"
server_key_iv: "6199911f9e1ea92987b9c4acc0fd5a1a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.697160 27638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:32:31.699913 27652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.701224 27638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:32:31.701350 27638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal
uuid: "ff77ab3625bb43488ef3bd605669c765"
format_stamp: "Formatted at 2026-09-26 12:32:31 on dist-test-slave-3jw2"
server_key: "eb5c830bcc7b4f27e85069f3b6aa8bd2"
server_key_iv: "6199911f9e1ea92987b9c4acc0fd5a1a"
server_key_version: "encryptionkey@0"
I20260926 12:32:31.701483 27638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:31.758932 27638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:31.759723 27638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:31.759953 27638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:31.760653 27638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:32:31.761668 27638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:32:31.761741 27638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.761816 27638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:32:31.761868 27638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:31.771560 27638 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.3:36217
I20260926 12:32:31.771587 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.3:36217 every 8 connection(s)
I20260926 12:32:31.772548 27638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/data/info.pb
I20260926 12:32:31.776088 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27638
I20260926 12:32:31.776232 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal/instance
I20260926 12:32:31.776503 7564 external_mini_cluster.cc:1428] Setting key c176a921e651650dc27a43d99c80a1f8
I20260926 12:32:31.782590 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.774517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.3:53745 (local address 127.7.99.62:35787)
0926 12:32:31.774685 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.774690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.775804 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 12:32:31.777483 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.777488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.777549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.777669 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:31.778741 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.779353 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.780639 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.780865 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.780951 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.782109 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:31.782112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:31.782114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:31.782127 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:31.782155 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:31.782163 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:31.782229 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.782394 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.782449 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:32:31.783506 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.774958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:35787 (local address 127.7.99.3:53745)
0926 12:32:31.775653 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.775695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.777248 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.777858 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.777865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.777958 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:31.778532 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.778552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.779495 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.779498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.780500 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.780511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.781006 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.781009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.781177 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.781828 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:32:31.781830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:31.781875 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:32:31.781955 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:31.782261 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:31.782267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:31.782271 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.782377 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.782625 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260926 12:32:31.784773 27766 heartbeater.cc:347] Connected to a master server at 127.7.99.62:35787
I20260926 12:32:31.785032 27766 heartbeater.cc:464] Registering TS with master...
I20260926 12:32:31.785599 27766 heartbeater.cc:511] Master 127.7.99.62:35787 requested a full tablet report, sending...
I20260926 12:32:31.786722 27316 ts_manager.cc:194] Registered new tserver with Master: ff77ab3625bb43488ef3bd605669c765 (127.7.99.3:36217)
I20260926 12:32:31.787320 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.99.3:53745
I20260926 12:32:31.790632 7564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:32:31.796576 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.792696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.7.99.62:35787)
0926 12:32:31.792828 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.792832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.792982 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:32:31.793074 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.793076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.793114 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.793189 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:31.793519 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.794067 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.794940 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.795196 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.795296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.796086 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:31.796089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:31.796091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:31.796103 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:31.796120 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:31.796155 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:31.796300 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.796396 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.796450 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:32:31.799566 27316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:32:31.803303 27316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:32:31.818950 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.812971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34941 (local address 127.0.0.1:58588)
0926 12:32:31.813571 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.813592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.813677 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.814394 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.814396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.814411 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:31.814484 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.814487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.815736 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.815738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.817277 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.817286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.818381 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.818385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.818542 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.818593 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.818690 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.818785 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":45,"thread_start_us":96,"threads_started":1}
I20260926 12:32:31.819073 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.813192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.7.99.1:34941)
0926 12:32:31.813925 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.813931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.813954 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:32:31.814027 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.814032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.814260 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.814420 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:31.814548 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.815639 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.817407 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.818194 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.818512 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.818692 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.818746 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.818840 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":99,"threads_started":1}
I20260926 12:32:31.819741 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.812343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.2:39229 (local address 127.0.0.1:47980)
0926 12:32:31.813312 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.813327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.813416 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.813962 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.813968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.813991 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:31.814137 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.814149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.815572 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.815574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.817244 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.817254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.818371 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.818375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.819434 (+ 1059us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.819457 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.819527 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.819608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":52,"thread_start_us":102,"threads_started":1}
I20260926 12:32:31.820214 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.813035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.7.99.2:39229)
0926 12:32:31.813378 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.813390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.813417 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:32:31.813557 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.813563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.813800 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.813961 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:31.814264 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.815475 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.817378 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.818249 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.818364 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.818572 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.819914 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.820018 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
I20260926 12:32:31.820377 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36217 (local address 127.0.0.1:50054)
0926 12:32:31.812847 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.812888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.813054 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.816395 (+ 3341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.816398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.816419 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:31.816523 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.816530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.817755 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.817758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.818816 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.818826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.819951 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.819954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.820037 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.820060 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.820127 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.820212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":96,"threads_started":1}
I20260926 12:32:31.820914 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.812332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.7.99.3:36217)
0926 12:32:31.815879 (+ 3547us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.815888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.815908 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:32:31.815966 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.815969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.816189 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.816467 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:31.816626 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.817634 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.818943 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.819819 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.820544 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.820689 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.820728 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.820796 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3450,"thread_start_us":151,"threads_started":1}
I20260926 12:32:31.822201 27438 tablet_service.cc:1467] Processing CreateTablet for tablet 02877563c11c42eabe7d65808d06e9e2 (DEFAULT_TABLE table=test-table [id=700cfe9d6fd546e0b986f89bf863a897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:31.823387 27438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02877563c11c42eabe7d65808d06e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:31.823752 27700 tablet_service.cc:1467] Processing CreateTablet for tablet 02877563c11c42eabe7d65808d06e9e2 (DEFAULT_TABLE table=test-table [id=700cfe9d6fd546e0b986f89bf863a897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:31.824802 27700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02877563c11c42eabe7d65808d06e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:31.826613 27569 tablet_service.cc:1467] Processing CreateTablet for tablet 02877563c11c42eabe7d65808d06e9e2 (DEFAULT_TABLE table=test-table [id=700cfe9d6fd546e0b986f89bf863a897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:32:31.827473 27569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02877563c11c42eabe7d65808d06e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:31.831207 27783 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Bootstrap starting.
I20260926 12:32:31.832638 27784 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Bootstrap starting.
I20260926 12:32:31.833849 27783 tablet_bootstrap.cc:654] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:31.834704 27784 tablet_bootstrap.cc:654] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:31.834889 27783 log.cc:824] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:31.835444 27784 log.cc:824] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:31.835958 27785 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Bootstrap starting.
I20260926 12:32:31.837037 27783 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: No bootstrap required, opened a new log
I20260926 12:32:31.837252 27783 ts_tablet_manager.cc:1397] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 12:32:31.837476 27784 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: No bootstrap required, opened a new log
I20260926 12:32:31.837677 27784 ts_tablet_manager.cc:1397] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:32:31.837991 27785 tablet_bootstrap.cc:654] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:31.838670 27785 log.cc:824] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:31.840603 27785 tablet_bootstrap.cc:492] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: No bootstrap required, opened a new log
I20260926 12:32:31.840806 27785 ts_tablet_manager.cc:1397] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:32:31.841895 27784 raft_consensus.cc:348] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.841895 27783 raft_consensus.cc:348] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.842170 27783 raft_consensus.cc:374] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:31.842170 27784 raft_consensus.cc:374] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:31.842237 27784 raft_consensus.cc:729] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff77ab3625bb43488ef3bd605669c765, State: Initialized, Role: FOLLOWER
I20260926 12:32:31.842286 27783 raft_consensus.cc:729] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2992f00b14254ed087669c7356b266d0, State: Initialized, Role: FOLLOWER
I20260926 12:32:31.842725 27783 consensus_queue.cc:260] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.842736 27784 consensus_queue.cc:260] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.843633 27766 heartbeater.cc:503] Master 127.7.99.62:35787 was elected leader, sending a full tablet report...
I20260926 12:32:31.843747 27783 ts_tablet_manager.cc:1428] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:32:31.843756 27784 ts_tablet_manager.cc:1428] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:32:31.844491 27504 heartbeater.cc:503] Master 127.7.99.62:35787 was elected leader, sending a full tablet report...
I20260926 12:32:31.844806 27785 raft_consensus.cc:348] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.845073 27785 raft_consensus.cc:374] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:31.845147 27785 raft_consensus.cc:729] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b934ccf7d074bc788f90e12dd3e7e12, State: Initialized, Role: FOLLOWER
I20260926 12:32:31.845634 27785 consensus_queue.cc:260] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.846313 27635 heartbeater.cc:503] Master 127.7.99.62:35787 was elected leader, sending a full tablet report...
I20260926 12:32:31.846689 27785 ts_tablet_manager.cc:1428] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 12:32:31.848706 27505 tablet.cc:2369] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:31.892918 27791 raft_consensus.cc:482] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:32:31.893213 27791 raft_consensus.cc:504] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.894564 27791 leader_election.cc:288] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff77ab3625bb43488ef3bd605669c765 (127.7.99.3:36217), 2992f00b14254ed087669c7356b266d0 (127.7.99.1:34941)
I20260926 12:32:31.899070 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.894926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.3:36217 (local address 127.7.99.2:42107)
0926 12:32:31.895061 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.895102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.895229 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.895568 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.895573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.895599 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:31.895775 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.895787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.896991 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.896997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.897881 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.897890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.898605 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898616 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.898728 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.898747 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.898844 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.898921 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 12:32:31.899070 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.895038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:42107 (local address 127.7.99.3:36217)
0926 12:32:31.895234 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.895239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.895269 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:32:31.895334 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.895339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.895391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.895492 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:31.895930 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.896872 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898030 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.898462 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898605 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.898670 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.898859 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.898943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:32:31.899580 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.895368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.99.2:41649 (local address 127.7.99.1:34941)
0926 12:32:31.895567 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.895572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.895729 (+ 157us) server_negotiation.cc:407] Connection header received
0926 12:32:31.895857 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.895861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.895915 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.896006 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:32:31.896427 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.897176 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898307 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.898777 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898908 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.898968 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.899344 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.899445 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 12:32:31.899775 27720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02877563c11c42eabe7d65808d06e9e2" candidate_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77ab3625bb43488ef3bd605669c765" is_pre_election: true
I20260926 12:32:31.899964 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.895245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34941 (local address 127.7.99.2:41649)
0926 12:32:31.895620 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.895635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:31.895741 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:31.896049 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:31.896052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:31.896072 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:32:31.896252 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.896261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.897309 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.897315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.898171 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:31.898180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.898920 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.898924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:31.899035 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.899053 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:32:31.899738 (+ 685us) client_negotiation.cc:240] Negotiation successful
0926 12:32:31.899825 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":126,"threads_started":1}
I20260926 12:32:31.900139 27720 raft_consensus.cc:2454] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b934ccf7d074bc788f90e12dd3e7e12 in term 0.
I20260926 12:32:31.900673 27458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02877563c11c42eabe7d65808d06e9e2" candidate_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2992f00b14254ed087669c7356b266d0" is_pre_election: true
I20260926 12:32:31.900749 27524 leader_election.cc:302] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b934ccf7d074bc788f90e12dd3e7e12, ff77ab3625bb43488ef3bd605669c765; no voters:
I20260926 12:32:31.900949 27458 raft_consensus.cc:2454] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b934ccf7d074bc788f90e12dd3e7e12 in term 0.
I20260926 12:32:31.901091 27791 raft_consensus.cc:2781] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:32:31.901201 27791 raft_consensus.cc:482] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:32:31.901270 27791 raft_consensus.cc:3037] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:31.902376 27791 raft_consensus.cc:504] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.902733 27791 leader_election.cc:288] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [CANDIDATE]: Term 1 election: Requested vote from peers ff77ab3625bb43488ef3bd605669c765 (127.7.99.3:36217), 2992f00b14254ed087669c7356b266d0 (127.7.99.1:34941)
I20260926 12:32:31.903100 27720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02877563c11c42eabe7d65808d06e9e2" candidate_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77ab3625bb43488ef3bd605669c765"
I20260926 12:32:31.903229 27458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02877563c11c42eabe7d65808d06e9e2" candidate_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2992f00b14254ed087669c7356b266d0"
I20260926 12:32:31.903321 27720 raft_consensus.cc:3037] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:31.903404 27458 raft_consensus.cc:3037] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:31.904596 27720 raft_consensus.cc:2454] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b934ccf7d074bc788f90e12dd3e7e12 in term 1.
I20260926 12:32:31.904657 27458 raft_consensus.cc:2454] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b934ccf7d074bc788f90e12dd3e7e12 in term 1.
I20260926 12:32:31.904942 27524 leader_election.cc:302] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b934ccf7d074bc788f90e12dd3e7e12, ff77ab3625bb43488ef3bd605669c765; no voters:
I20260926 12:32:31.905114 27791 raft_consensus.cc:2781] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:31.905349 27791 raft_consensus.cc:686] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 1 LEADER]: Becoming Leader. State: Replica: 3b934ccf7d074bc788f90e12dd3e7e12, State: Running, Role: LEADER
I20260926 12:32:31.905802 27791 consensus_queue.cc:237] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } }
I20260926 12:32:31.909562 27315 catalog_manager.cc:5495] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b934ccf7d074bc788f90e12dd3e7e12 (127.7.99.2). New cstate: current_term: 1 leader_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b934ccf7d074bc788f90e12dd3e7e12" member_type: VOTER last_known_addr { host: "127.7.99.2" port: 39229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 } health_report { overall_health: UNKNOWN } } }
I20260926 12:32:31.924989 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.921517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.7.99.2:39229)
0926 12:32:31.921655 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:31.921658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:31.921673 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:32:31.921774 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:31.921777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:31.921832 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:31.921928 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:32:31.922174 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.922705 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.923552 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:31.923774 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:31.923858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:31.924123 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:32:31.924629 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:32:31.924721 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:31.924794 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 12:32:31.924842 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:32:31.931396 27720 raft_consensus.cc:1261] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 1 FOLLOWER]: Refusing update from remote peer 3b934ccf7d074bc788f90e12dd3e7e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:31.931442 27458 raft_consensus.cc:1261] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 1 FOLLOWER]: Refusing update from remote peer 3b934ccf7d074bc788f90e12dd3e7e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:31.932268 27793 consensus_queue.cc:1035] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2992f00b14254ed087669c7356b266d0" member_type: VOTER last_known_addr { host: "127.7.99.1" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:31.932483 27791 consensus_queue.cc:1035] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff77ab3625bb43488ef3bd605669c765" member_type: VOTER last_known_addr { host: "127.7.99.3" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:31.940423 27797 mvcc.cc:205] Tried to move back new op lower bound from 7333584699102547968 to 7333584699009220608. Current Snapshot: MvccSnapshot[applied={T|T < 7333584699102547968}]
I20260926 12:32:31.948833 27315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:31.949016 27315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:32:31.951475 27315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f54f18b5d6d94745b8c5ca7f12a8ef19: Sending DeleteTablet for 3 replicas of tablet 02877563c11c42eabe7d65808d06e9e2
I20260926 12:32:31.952462 27438 tablet_service.cc:1514] Processing DeleteTablet for tablet 02877563c11c42eabe7d65808d06e9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:31 UTC) from {username='slave'} at 127.0.0.1:58588
I20260926 12:32:31.952440 27700 tablet_service.cc:1514] Processing DeleteTablet for tablet 02877563c11c42eabe7d65808d06e9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:31 UTC) from {username='slave'} at 127.0.0.1:50054
I20260926 12:32:31.953016 27806 tablet_replica.cc:330] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: stopping tablet replica
I20260926 12:32:31.953022 27569 tablet_service.cc:1514] Processing DeleteTablet for tablet 02877563c11c42eabe7d65808d06e9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:32:31 UTC) from {username='slave'} at 127.0.0.1:47980
I20260926 12:32:31.953356 27807 tablet_replica.cc:330] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0: stopping tablet replica
I20260926 12:32:31.953454 27808 tablet_replica.cc:330] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: stopping tablet replica
I20260926 12:32:31.953603 27806 raft_consensus.cc:2229] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:31.953673 27807 raft_consensus.cc:2229] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:32:31.953730 27808 raft_consensus.cc:2229] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:32:31.953753 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27376
I20260926 12:32:31.954028 27806 raft_consensus.cc:2258] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:31.954030 27807 raft_consensus.cc:2258] T 02877563c11c42eabe7d65808d06e9e2 P 2992f00b14254ed087669c7356b266d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:31.954108 27808 raft_consensus.cc:2258] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:32:31.955678 27808 ts_tablet_manager.cc:1905] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:31.956058 27806 ts_tablet_manager.cc:1905] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:32:31.958586 27808 ts_tablet_manager.cc:1918] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:31.958706 27808 log.cc:1196] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-1/wal/wals/02877563c11c42eabe7d65808d06e9e2
I20260926 12:32:31.958866 27806 ts_tablet_manager.cc:1918] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:32:31.959018 27806 log.cc:1196] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Deleting WAL directory at /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425700812969-7564-0/minicluster-data/ts-2/wal/wals/02877563c11c42eabe7d65808d06e9e2
I20260926 12:32:31.959044 27808 ts_tablet_manager.cc:1939] T 02877563c11c42eabe7d65808d06e9e2 P 3b934ccf7d074bc788f90e12dd3e7e12: Deleting consensus metadata
I20260926 12:32:31.959388 27806 ts_tablet_manager.cc:1939] T 02877563c11c42eabe7d65808d06e9e2 P ff77ab3625bb43488ef3bd605669c765: Deleting consensus metadata
I20260926 12:32:31.960327 27303 catalog_manager.cc:4841] TS 3b934ccf7d074bc788f90e12dd3e7e12 (127.7.99.2:39229): tablet 02877563c11c42eabe7d65808d06e9e2 (table test-table [id=700cfe9d6fd546e0b986f89bf863a897]) successfully deleted
I20260926 12:32:31.960505 27302 catalog_manager.cc:4841] TS ff77ab3625bb43488ef3bd605669c765 (127.7.99.3:36217): tablet 02877563c11c42eabe7d65808d06e9e2 (table test-table [id=700cfe9d6fd546e0b986f89bf863a897]) successfully deleted
W20260926 12:32:31.962333 27302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.99.1:34941 (error 108)
I20260926 12:32:31.962553 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27507
I20260926 12:32:31.963600 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:31.963323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.1:34941 (local address 127.0.0.1:58592)
0926 12:32:31.963458 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:31.963529 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 12:32:31.963829 27302 catalog_manager.cc:4584] TS 2992f00b14254ed087669c7356b266d0 (127.7.99.1:34941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02877563c11c42eabe7d65808d06e9e2: Network error: Client connection negotiation failed: client connection to 127.7.99.1:34941: connect: Connection refused (error 111)
I20260926 12:32:31.967844 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27638
I20260926 12:32:31.973448 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27285
2026-09-26T12:32:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1083 ms)
[----------] 6 tests from EncryptionPolicyTest (10584 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:32 dist-test-slave-3jw2 krb5kdc[27812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:32 dist-test-slave-3jw2 krb5kdc[27812](info): set up 2 sockets
Sep 26 12:32:32 dist-test-slave-3jw2 krb5kdc[27812](info): commencing operation
krb5kdc: starting...
W20260926 12:32:33.032476 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:33.153249 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:33.263851 27825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:33.264161 27825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:33.264264 27825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:33.268280 27825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:33.268393 27825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:33.268436 27825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:33.268499 27825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:33.268537 27825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:33.273900 27825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:33.275029 27825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:33.276113 27825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:33.284530 27831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:33.284675 27825 server_base.cc:1034] running on GCE node
W20260926 12:32:33.284505 27830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:33.284548 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:33.285341 27825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:33.286412 27825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:33.287603 27825 hybrid_clock.cc:648] HybridClock initialized: now 1790425953287584 us; error 31 us; skew 500 ppm
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:33.294746 27825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:33.295706 27825 webserver.cc:460] Webserver started at http://127.0.0.1:43737/ using document root <none> and password file <none>
I20260926 12:32:33.296368 27825 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:33.296437 27825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:33.296666 27825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:33.298911 27825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8"
format_stamp: "Formatted at 2026-09-26 12:32:33 on dist-test-slave-3jw2"
server_key: "808af42e54f540122220ee21a1a83219"
server_key_iv: "4c11f2a2337ee10043bb083b016693c3"
server_key_version: "encryptionkey@0"
I20260926 12:32:33.299507 27825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8"
format_stamp: "Formatted at 2026-09-26 12:32:33 on dist-test-slave-3jw2"
server_key: "808af42e54f540122220ee21a1a83219"
server_key_iv: "4c11f2a2337ee10043bb083b016693c3"
server_key_version: "encryptionkey@0"
I20260926 12:32:33.303319 27825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 12:32:33.305825 27840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:33.307078 27825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:33.307215 27825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8"
format_stamp: "Formatted at 2026-09-26 12:32:33 on dist-test-slave-3jw2"
server_key: "808af42e54f540122220ee21a1a83219"
server_key_iv: "4c11f2a2337ee10043bb083b016693c3"
server_key_version: "encryptionkey@0"
I20260926 12:32:33.307327 27825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:33.323999 27825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:33.326727 27825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:33.326931 27825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:33.334295 27825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35775
I20260926 12:32:33.334324 27892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35775 every 8 connection(s)
I20260926 12:32:33.335229 27825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:33.337842 27893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:33.339488 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27825
I20260926 12:32:33.339629 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:33.339921 7564 external_mini_cluster.cc:1428] Setting key aaa0de047edf6a38080ac40b8b821833
I20260926 12:32:33.343878 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8: Bootstrap starting.
I20260926 12:32:33.346009 27893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:33.346698 27893 log.cc:824] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:32:33.348680 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8: No bootstrap required, opened a new log
I20260926 12:32:33.352587 27893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } }
I20260926 12:32:33.352875 27893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:33.352910 27893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f833e89dbd449ec8596a0b6bfaa4ca8, State: Initialized, Role: FOLLOWER
I20260926 12:32:33.353396 27893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } }
I20260926 12:32:33.353550 27893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:33.353646 27893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:33.353776 27893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:33.354667 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:33.341374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47947 (local address 127.0.0.1:35775)
0926 12:32:33.341810 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:33.341824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:33.341860 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:32:33.342464 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:33.342479 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:33.342806 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:33.343143 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:33.343950 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:33.345038 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:33.345899 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:33.346427 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:33.346586 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:33.349408 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:33.349413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:33.349426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:33.349451 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:33.351322 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:33.351927 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:33.351933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:33.351938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:33.352059 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:33.352563 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:33.352566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:33.352568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:33.353134 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:33.353399 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:33.353696 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 12:32:33.353930 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260926 12:32:33.354689 27893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } }
I20260926 12:32:33.355046 27893 leader_election.cc:302] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f833e89dbd449ec8596a0b6bfaa4ca8; no voters:
I20260926 12:32:33.355368 27893 leader_election.cc:288] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:33.355459 27898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:33.355688 27898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 3f833e89dbd449ec8596a0b6bfaa4ca8, State: Running, Role: LEADER
I20260926 12:32:33.356021 27898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } }
I20260926 12:32:33.356666 27893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:33.357719 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f833e89dbd449ec8596a0b6bfaa4ca8. Latest consensus state: current_term: 1 leader_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } } }
I20260926 12:32:33.357863 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:33.358002 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:33.358217 27899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f833e89dbd449ec8596a0b6bfaa4ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35775 } } }
I20260926 12:32:33.358299 27899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:33.359372 27907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:33.362663 27907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:33.368847 27907 catalog_manager.cc:1347] Generated new cluster ID: 224d60bc45d748ffbf71727811b97b29
I20260926 12:32:33.369223 27907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:33.371853 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:33.360422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46239 (local address 127.0.0.1:35775)
0926 12:32:33.360543 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:33.360547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:33.363063 (+ 2516us) server_negotiation.cc:407] Connection header received
0926 12:32:33.363172 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:33.363175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:33.363228 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:33.363328 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:33.364120 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:33.364920 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:33.366019 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:33.366485 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:33.366622 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:33.368068 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:33.368072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:33.368075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:33.368102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:33.369884 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:33.370481 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:33.370485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:33.370487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:33.370565 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:33.370932 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:33.370935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:33.370937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:33.371114 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:33.371246 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:33.371558 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 12:32:33.371658 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:32:33.394723 27907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:33.395707 27907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:33.401595 27907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f833e89dbd449ec8596a0b6bfaa4ca8: Generated new TSK 0
I20260926 12:32:33.402240 27907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:33.410898 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27825
W20260926 12:32:33.415818 27810 connection.cc:535] client connection to 127.0.0.1:35775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35775): Connection reset by peer (error 104)
2026-09-26T12:32:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27924](info): set up 2 sockets
Sep 26 12:32:33 dist-test-slave-3jw2 krb5kdc[27924](info): commencing operation
krb5kdc: starting...
W20260926 12:32:34.452656 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:34 dist-test-slave-3jw2 krb5kdc[27924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:34.571522 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:34.678570 27937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:34.678979 27937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:34.679102 27937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:34.683382 27937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:34.683477 27937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:34.683531 27937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:34.683612 27937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:34.683652 27937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:34.688819 27937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:34.689915 27937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:34.690945 27937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:32:34.699514 27937 server_base.cc:1034] running on GCE node
W20260926 12:32:34.699479 27943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:34.699479 27942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:34.699507 27945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:34.700320 27937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:34.701363 27937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:34.702522 27937 hybrid_clock.cc:648] HybridClock initialized: now 1790425954702504 us; error 31 us; skew 500 ppm
Sep 26 12:32:34 dist-test-slave-3jw2 krb5kdc[27924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:34.705269 27937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:34.706079 27937 webserver.cc:460] Webserver started at http://127.0.0.1:41781/ using document root <none> and password file <none>
I20260926 12:32:34.706641 27937 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:34.706728 27937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:34.706964 27937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:34.709991 27937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "b4f571d49262453b9a8aff43b354fc07"
format_stamp: "Formatted at 2026-09-26 12:32:34 on dist-test-slave-3jw2"
server_key: "0be4e4ab58e6a9299da6bbac733da440"
server_key_iv: "bd1126415446300fe8cff98a3027eec8"
server_key_version: "encryptionkey@0"
I20260926 12:32:34.710675 27937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "b4f571d49262453b9a8aff43b354fc07"
format_stamp: "Formatted at 2026-09-26 12:32:34 on dist-test-slave-3jw2"
server_key: "0be4e4ab58e6a9299da6bbac733da440"
server_key_iv: "bd1126415446300fe8cff98a3027eec8"
server_key_version: "encryptionkey@0"
I20260926 12:32:34.715292 27937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:32:34.718195 27952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:34.719430 27937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:34.719570 27937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "b4f571d49262453b9a8aff43b354fc07"
format_stamp: "Formatted at 2026-09-26 12:32:34 on dist-test-slave-3jw2"
server_key: "0be4e4ab58e6a9299da6bbac733da440"
server_key_iv: "bd1126415446300fe8cff98a3027eec8"
server_key_version: "encryptionkey@0"
I20260926 12:32:34.719682 27937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:34.734894 27937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:34.738145 27937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:34.738344 27937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:34.745229 27937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46393
I20260926 12:32:34.745260 28004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46393 every 8 connection(s)
I20260926 12:32:34.746150 27937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:34.747184 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 27937
I20260926 12:32:34.747323 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:34.747604 7564 external_mini_cluster.cc:1428] Setting key 21cece8172cc8303b78c918659178e6a
I20260926 12:32:34.749188 28005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:34.755690 28005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07: Bootstrap starting.
I20260926 12:32:34.757166 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:34.748862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60019 (local address 127.0.0.1:46393)
0926 12:32:34.749263 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:34.749289 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:34.749330 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:32:34.750132 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:34.750152 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:34.750559 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:34.750895 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:34.751271 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:34.752756 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:34.753628 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:34.754309 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:34.754466 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:34.755274 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:34.755288 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:34.755303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:34.755770 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:34.755833 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:34.755862 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:34.755988 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:34.756147 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 12:32:34.756302 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260926 12:32:34.758273 28005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:34.759069 28005 log.cc:824] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:34.760003 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:34.760953 28005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07: No bootstrap required, opened a new log
I20260926 12:32:34.764359 28005 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260926 12:32:34.764622 28005 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:34.764688 28005 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f571d49262453b9a8aff43b354fc07, State: Initialized, Role: FOLLOWER
I20260926 12:32:34.765172 28005 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260926 12:32:34.765367 28005 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:34.765436 28005 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:34.765532 28005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:34.766384 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:34.761931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48347 (local address 127.0.0.1:46393)
0926 12:32:34.762050 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:34.762054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:34.762228 (+ 174us) server_negotiation.cc:407] Connection header received
0926 12:32:34.762331 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:34.762333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:34.762366 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:34.762426 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:34.762690 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:34.763510 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:34.764636 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:34.764964 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:34.765042 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:34.765907 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:34.765909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:34.765911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:34.765921 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:34.765943 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:34.765952 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:34.766018 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:34.766176 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 12:32:34.766226 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:32:34.766652 28005 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260926 12:32:34.766990 28005 leader_election.cc:302] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f571d49262453b9a8aff43b354fc07; no voters:
I20260926 12:32:34.767318 28005 leader_election.cc:288] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:34.767376 28015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:34.767583 28015 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [term 1 LEADER]: Becoming Leader. State: Replica: b4f571d49262453b9a8aff43b354fc07, State: Running, Role: LEADER
I20260926 12:32:34.767901 28015 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260926 12:32:34.768576 28005 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:34.769641 28017 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f571d49262453b9a8aff43b354fc07. Latest consensus state: current_term: 1 leader_uuid: "b4f571d49262453b9a8aff43b354fc07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260926 12:32:34.769661 28016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f571d49262453b9a8aff43b354fc07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f571d49262453b9a8aff43b354fc07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260926 12:32:34.769863 28016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:34.770146 28017 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:34.770313 28024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:34.772989 28024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:34.778103 28024 catalog_manager.cc:1347] Generated new cluster ID: 96a4c176555f49eb8a7b13a66d4407bc
I20260926 12:32:34.778167 28024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:34.789160 28024 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:34.790031 28024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:34.795600 28024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f571d49262453b9a8aff43b354fc07: Generated new TSK 0
I20260926 12:32:34.796331 28024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:34.800835 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 27937
W20260926 12:32:34.806113 27922 connection.cc:535] client connection to 127.0.0.1:46393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46393): Connection reset by peer (error 104)
2026-09-26T12:32:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:34 dist-test-slave-3jw2 krb5kdc[28036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:34 dist-test-slave-3jw2 krb5kdc[28036](info): set up 2 sockets
Sep 26 12:32:34 dist-test-slave-3jw2 krb5kdc[28036](info): commencing operation
krb5kdc: starting...
W20260926 12:32:35.836658 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:35 dist-test-slave-3jw2 krb5kdc[28036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:35.954938 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:36.062023 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:36.062331 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:36.062422 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:36.066387 28049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:36.066476 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:36.066514 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:36.066550 28049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:36.066581 28049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:36.071682 28049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:36.072721 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:36.073716 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:36.081878 28054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:36.082002 28057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:36.082062 28055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:36.082002 28049 server_base.cc:1034] running on GCE node
I20260926 12:32:36.082667 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:36.083658 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:36.084838 28049 hybrid_clock.cc:648] HybridClock initialized: now 1790425956084833 us; error 30 us; skew 500 ppm
Sep 26 12:32:36 dist-test-slave-3jw2 krb5kdc[28036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:36.087327 28049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:36.088151 28049 webserver.cc:460] Webserver started at http://127.0.0.1:46849/ using document root <none> and password file <none>
I20260926 12:32:36.088727 28049 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:36.088828 28049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:36.089048 28049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:36.090956 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "d42c4683a9c344f19ade08ea1dc528bd"
format_stamp: "Formatted at 2026-09-26 12:32:36 on dist-test-slave-3jw2"
server_key: "cd35ff7b7f0b1f7bfa2c05682ad45884"
server_key_iv: "f52ae32b24377a4a87cca3cbc23d02bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:36.091482 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "d42c4683a9c344f19ade08ea1dc528bd"
format_stamp: "Formatted at 2026-09-26 12:32:36 on dist-test-slave-3jw2"
server_key: "cd35ff7b7f0b1f7bfa2c05682ad45884"
server_key_iv: "f52ae32b24377a4a87cca3cbc23d02bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:36.094995 28049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:32:36.097267 28064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:36.098389 28049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:36.098520 28049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "d42c4683a9c344f19ade08ea1dc528bd"
format_stamp: "Formatted at 2026-09-26 12:32:36 on dist-test-slave-3jw2"
server_key: "cd35ff7b7f0b1f7bfa2c05682ad45884"
server_key_iv: "f52ae32b24377a4a87cca3cbc23d02bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:36.098627 28049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:36.102969 28049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:36.105669 28049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:36.105854 28049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:36.112586 28049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40671
I20260926 12:32:36.112612 28116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40671 every 8 connection(s)
I20260926 12:32:36.113435 28049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:36.115906 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:36.120435 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28049
I20260926 12:32:36.120596 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:36.120910 7564 external_mini_cluster.cc:1428] Setting key e71fd55155213551d0062f4200fe72ae
I20260926 12:32:36.121784 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd: Bootstrap starting.
I20260926 12:32:36.124015 28117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:36.124892 28117 log.cc:824] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:36.126473 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:36.122222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52761 (local address 127.0.0.1:40671)
0926 12:32:36.122635 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:36.122648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:36.122681 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:32:36.123449 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:36.123465 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:36.123740 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:36.124065 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:36.124997 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:36.125001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:36.125036 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:36.125047 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:36.125105 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:36.125124 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:36.125238 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:36.125356 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 12:32:36.125511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260926 12:32:36.127018 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd: No bootstrap required, opened a new log
I20260926 12:32:36.129156 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:36.130781 28117 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260926 12:32:36.131093 28117 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:36.131192 28117 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42c4683a9c344f19ade08ea1dc528bd, State: Initialized, Role: FOLLOWER
I20260926 12:32:36.131691 28117 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260926 12:32:36.131867 28117 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:36.131944 28117 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:36.132059 28117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:36.132710 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:36.131105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46667 (local address 127.0.0.1:40671)
0926 12:32:36.131211 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:36.131215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:36.131337 (+ 122us) server_negotiation.cc:407] Connection header received
0926 12:32:36.131497 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:36.131500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:36.131530 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:36.131598 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:36.132364 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:36.132371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:36.132377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:36.132390 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:36.132417 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:36.132431 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:36.132516 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:36.132577 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:32:36.132609 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:32:36.133178 28117 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260926 12:32:36.133507 28117 leader_election.cc:302] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42c4683a9c344f19ade08ea1dc528bd; no voters:
I20260926 12:32:36.133790 28117 leader_election.cc:288] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:36.133852 28127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:36.134090 28127 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [term 1 LEADER]: Becoming Leader. State: Replica: d42c4683a9c344f19ade08ea1dc528bd, State: Running, Role: LEADER
I20260926 12:32:36.134420 28127 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260926 12:32:36.134950 28117 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:36.136202 28128 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42c4683a9c344f19ade08ea1dc528bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } } }
I20260926 12:32:36.136252 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42c4683a9c344f19ade08ea1dc528bd. Latest consensus state: current_term: 1 leader_uuid: "d42c4683a9c344f19ade08ea1dc528bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42c4683a9c344f19ade08ea1dc528bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } } }
I20260926 12:32:36.136322 28128 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:36.136344 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:36.136669 28136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:36.139256 28136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:36.144541 28136 catalog_manager.cc:1347] Generated new cluster ID: 7ceb1cf44f1e423e999f1dadeddfec26
I20260926 12:32:36.144605 28136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:36.168198 28136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:36.168538 28136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:36.173017 28136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42c4683a9c344f19ade08ea1dc528bd: Generated new TSK 0
I20260926 12:32:36.173717 28136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:36.184330 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28049
2026-09-26T12:32:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:36 dist-test-slave-3jw2 krb5kdc[28148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:36 dist-test-slave-3jw2 krb5kdc[28148](info): set up 2 sockets
Sep 26 12:32:36 dist-test-slave-3jw2 krb5kdc[28148](info): commencing operation
krb5kdc: starting...
W20260926 12:32:37.268496 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:37.387557 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:37.494421 28161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:37.494649 28161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:37.494688 28161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:37.498832 28161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:37.498883 28161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:37.498901 28161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:37.498917 28161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:37.498960 28161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:37.504040 28161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:37.505025 28161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:37.506014 28161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:37.514303 28167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:37.514441 28161 server_base.cc:1034] running on GCE node
W20260926 12:32:37.514287 28169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:37.514359 28166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:37.515147 28161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:37.516245 28161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:37.517432 28161 hybrid_clock.cc:648] HybridClock initialized: now 1790425957517420 us; error 30 us; skew 500 ppm
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:37.520229 28161 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:37.521070 28161 webserver.cc:460] Webserver started at http://127.0.0.1:42081/ using document root <none> and password file <none>
I20260926 12:32:37.521608 28161 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:37.521697 28161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:37.521936 28161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:37.524145 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "cbd26c29879042c499cf13251d2e9df7"
format_stamp: "Formatted at 2026-09-26 12:32:37 on dist-test-slave-3jw2"
server_key: "f64dfbdc76cb069f600ee5b8870294e8"
server_key_iv: "dd962d48a79947104eded67f0f9ec311"
server_key_version: "encryptionkey@0"
I20260926 12:32:37.524827 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "cbd26c29879042c499cf13251d2e9df7"
format_stamp: "Formatted at 2026-09-26 12:32:37 on dist-test-slave-3jw2"
server_key: "f64dfbdc76cb069f600ee5b8870294e8"
server_key_iv: "dd962d48a79947104eded67f0f9ec311"
server_key_version: "encryptionkey@0"
I20260926 12:32:37.528653 28161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:32:37.531028 28176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:37.532246 28161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:37.532377 28161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "cbd26c29879042c499cf13251d2e9df7"
format_stamp: "Formatted at 2026-09-26 12:32:37 on dist-test-slave-3jw2"
server_key: "f64dfbdc76cb069f600ee5b8870294e8"
server_key_iv: "dd962d48a79947104eded67f0f9ec311"
server_key_version: "encryptionkey@0"
I20260926 12:32:37.532486 28161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:37.542436 28161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:37.545167 28161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:37.545306 28161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:37.551836 28161 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34189
I20260926 12:32:37.551924 28228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34189 every 8 connection(s)
I20260926 12:32:37.552912 28161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:37.553136 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28161
I20260926 12:32:37.553285 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:37.553628 7564 external_mini_cluster.cc:1428] Setting key dc67d1f65ce12cb54a24cf92ad28bec2
I20260926 12:32:37.555891 28229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:37.562031 28229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7: Bootstrap starting.
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:32:37.564558 28229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:37.565436 28229 log.cc:824] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:37.567559 28229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7: No bootstrap required, opened a new log
I20260926 12:32:37.567688 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:37.554894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41051 (local address 127.0.0.1:34189)
0926 12:32:37.555291 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:37.555307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:37.555341 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:37.555948 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:37.555962 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:37.556411 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:37.556681 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:37.557550 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:37.558796 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:37.559645 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:37.560106 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:37.560328 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:37.562991 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:37.562996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:37.563007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:37.563032 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:37.564804 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:37.565258 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:37.565262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:37.565267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:37.565371 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:37.565662 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:37.565664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:37.565666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:37.566045 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:37.566235 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:37.566507 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 12:32:37.566789 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260926 12:32:37.570307 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:37.571890 28229 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260926 12:32:37.572232 28229 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:37.572341 28229 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd26c29879042c499cf13251d2e9df7, State: Initialized, Role: FOLLOWER
I20260926 12:32:37.572836 28229 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260926 12:32:37.573009 28229 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:37.573081 28229 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:37.573194 28229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:37.574146 28229 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260926 12:32:37.574494 28229 leader_election.cc:302] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd26c29879042c499cf13251d2e9df7; no voters:
I20260926 12:32:37.574805 28229 leader_election.cc:288] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:37.574916 28239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:37.575152 28239 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [term 1 LEADER]: Becoming Leader. State: Replica: cbd26c29879042c499cf13251d2e9df7, State: Running, Role: LEADER
I20260926 12:32:37.575533 28239 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260926 12:32:37.575964 28229 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:37.577248 28241 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd26c29879042c499cf13251d2e9df7. Latest consensus state: current_term: 1 leader_uuid: "cbd26c29879042c499cf13251d2e9df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260926 12:32:37.577378 28241 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:37.577713 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbd26c29879042c499cf13251d2e9df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd26c29879042c499cf13251d2e9df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260926 12:32:37.577809 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:37.579167 28249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:37.580861 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:37.572250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49711 (local address 127.0.0.1:34189)
0926 12:32:37.572354 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:37.572358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:37.572510 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:32:37.572606 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:37.572610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:37.572654 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:37.572732 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:37.573548 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:37.574027 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:37.575137 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:37.575481 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:37.575580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:37.577391 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:37.577396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:37.577399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:37.577424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:37.579179 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:37.579583 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:37.579587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:37.579589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:37.579673 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:37.579901 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:37.579904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:37.579906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:37.580131 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:37.580288 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:37.580556 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:32:37.580696 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:32:37.582643 28249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:37.588241 28249 catalog_manager.cc:1347] Generated new cluster ID: 5209181a529248c8ba6d3323faae7d2e
I20260926 12:32:37.588318 28249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:37.596325 28249 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:37.597211 28249 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:37.602442 28249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbd26c29879042c499cf13251d2e9df7: Generated new TSK 0
I20260926 12:32:37.603057 28249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:37.609972 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28161
2026-09-26T12:32:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28260](info): set up 2 sockets
Sep 26 12:32:37 dist-test-slave-3jw2 krb5kdc[28260](info): commencing operation
krb5kdc: starting...
W20260926 12:32:38.660588 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:38 dist-test-slave-3jw2 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:38.783187 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:38.891649 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:38.891906 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:38.891948 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:38.895926 28273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:38.895982 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:38.896000 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:38.896015 28273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:38.896030 28273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:38.901392 28273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:38.902367 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:38.903302 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:38.911705 28281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:38.911725 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:38.911681 28279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:38.911846 28273 server_base.cc:1034] running on GCE node
I20260926 12:32:38.912477 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:38.913537 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:38.914696 28273 hybrid_clock.cc:648] HybridClock initialized: now 1790425958914685 us; error 27 us; skew 500 ppm
Sep 26 12:32:38 dist-test-slave-3jw2 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:38.922565 28273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:38.923444 28273 webserver.cc:460] Webserver started at http://127.0.0.1:40071/ using document root <none> and password file <none>
I20260926 12:32:38.924007 28273 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:38.924067 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:38.924387 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:38.926349 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23"
format_stamp: "Formatted at 2026-09-26 12:32:38 on dist-test-slave-3jw2"
server_key: "9a9f9d4ed0d9b79bd5a2f8205b9a57cf"
server_key_iv: "ba33538973cc0d943e07c4246c74b170"
server_key_version: "encryptionkey@0"
I20260926 12:32:38.926916 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23"
format_stamp: "Formatted at 2026-09-26 12:32:38 on dist-test-slave-3jw2"
server_key: "9a9f9d4ed0d9b79bd5a2f8205b9a57cf"
server_key_iv: "ba33538973cc0d943e07c4246c74b170"
server_key_version: "encryptionkey@0"
I20260926 12:32:38.930505 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:32:38.932809 28288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:38.933974 28273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:32:38.934120 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23"
format_stamp: "Formatted at 2026-09-26 12:32:38 on dist-test-slave-3jw2"
server_key: "9a9f9d4ed0d9b79bd5a2f8205b9a57cf"
server_key_iv: "ba33538973cc0d943e07c4246c74b170"
server_key_version: "encryptionkey@0"
I20260926 12:32:38.934244 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:38.962793 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:38.965667 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:38.965900 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:38.972847 28273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40539
I20260926 12:32:38.972870 28340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40539 every 8 connection(s)
I20260926 12:32:38.973753 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:38.976416 28341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:38.979489 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28273
I20260926 12:32:38.979628 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:38.979943 7564 external_mini_cluster.cc:1428] Setting key b0b5b764faf39db1ff88d20a71b07de5
I20260926 12:32:38.982272 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23: Bootstrap starting.
I20260926 12:32:38.984413 28341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:38.985112 28341 log.cc:824] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:38.986830 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23: No bootstrap required, opened a new log
I20260926 12:32:38.989008 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:38.981333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33311 (local address 127.0.0.1:40539)
0926 12:32:38.981768 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:38.981786 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:38.981821 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:32:38.982482 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:38.982499 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:38.982843 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:38.983162 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:38.983560 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:38.984665 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:38.985638 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:38.986103 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:38.986276 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:38.987178 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:38.987183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:38.987196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:38.987563 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:38.987624 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:38.987646 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:38.987757 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:38.987871 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 12:32:38.988076 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
I20260926 12:32:38.990079 28341 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260926 12:32:38.990319 28341 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:38.990396 28341 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d10f8a6168431ba9b8fe06e2ad4b23, State: Initialized, Role: FOLLOWER
I20260926 12:32:38.990778 28341 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260926 12:32:38.990952 28341 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:38.991022 28341 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:38.991109 28341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:38.992307 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:38.992308 28341 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260926 12:32:38.992638 28341 leader_election.cc:302] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4d10f8a6168431ba9b8fe06e2ad4b23; no voters:
I20260926 12:32:38.992923 28341 leader_election.cc:288] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:38.993031 28346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:38.993237 28346 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [term 1 LEADER]: Becoming Leader. State: Replica: f4d10f8a6168431ba9b8fe06e2ad4b23, State: Running, Role: LEADER
I20260926 12:32:38.993642 28346 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260926 12:32:38.994100 28341 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:38.994832 28346 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4d10f8a6168431ba9b8fe06e2ad4b23. Latest consensus state: current_term: 1 leader_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } } }
I20260926 12:32:38.995008 28346 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:38.995301 28347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d10f8a6168431ba9b8fe06e2ad4b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } } }
I20260926 12:32:38.995389 28359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:38.995471 28347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:38.998543 28359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:39.000938 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:38.994556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60369 (local address 127.0.0.1:40539)
0926 12:32:38.994682 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:38.994686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:38.996454 (+ 1768us) server_negotiation.cc:407] Connection header received
0926 12:32:38.996560 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:38.996563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:38.996610 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:38.996691 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:38.996963 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:38.997678 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:38.999184 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:38.999500 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:38.999607 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:39.000502 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:39.000506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:39.000508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:39.000520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:39.000553 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:39.000564 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:39.000651 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:39.000736 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:32:39.000784 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:32:39.003925 28359 catalog_manager.cc:1347] Generated new cluster ID: 939ac4bba05f4b41b894b0163d6f7b0e
I20260926 12:32:39.003985 28359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:39.019052 28359 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:39.020133 28359 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:39.026731 28359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4d10f8a6168431ba9b8fe06e2ad4b23: Generated new TSK 0
I20260926 12:32:39.027498 28359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:39.036369 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28273
2026-09-26T12:32:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:39 dist-test-slave-3jw2 krb5kdc[28372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:39 dist-test-slave-3jw2 krb5kdc[28372](info): set up 2 sockets
Sep 26 12:32:39 dist-test-slave-3jw2 krb5kdc[28372](info): commencing operation
krb5kdc: starting...
W20260926 12:32:40.088443 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:40 dist-test-slave-3jw2 krb5kdc[28372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:40.207780 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:40.315616 28385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:40.315886 28385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:40.315934 28385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:40.319891 28385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:40.319981 28385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:40.320000 28385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:40.320017 28385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:40.320031 28385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:40.325601 28385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:40.326694 28385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:40.327692 28385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:40.336300 28391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:40.336293 28390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:40.336346 28385 server_base.cc:1034] running on GCE node
W20260926 12:32:40.336304 28393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:40.337008 28385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:40.338150 28385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:40.339316 28385 hybrid_clock.cc:648] HybridClock initialized: now 1790425960339295 us; error 29 us; skew 500 ppm
Sep 26 12:32:40 dist-test-slave-3jw2 krb5kdc[28372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:40.342005 28385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:40.342947 28385 webserver.cc:460] Webserver started at http://127.0.0.1:36195/ using document root <none> and password file <none>
I20260926 12:32:40.343497 28385 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:40.343555 28385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:40.343786 28385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:40.345841 28385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "95424705e01f48328075ead4b0e8b6b2"
format_stamp: "Formatted at 2026-09-26 12:32:40 on dist-test-slave-3jw2"
server_key: "503340313cfd143121ef996b8777651e"
server_key_iv: "a0ae3412a33b208d7dfc1fcbbd737935"
server_key_version: "encryptionkey@0"
I20260926 12:32:40.346413 28385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "95424705e01f48328075ead4b0e8b6b2"
format_stamp: "Formatted at 2026-09-26 12:32:40 on dist-test-slave-3jw2"
server_key: "503340313cfd143121ef996b8777651e"
server_key_iv: "a0ae3412a33b208d7dfc1fcbbd737935"
server_key_version: "encryptionkey@0"
I20260926 12:32:40.350056 28385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 12:32:40.352478 28400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:40.353775 28385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:32:40.353927 28385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "95424705e01f48328075ead4b0e8b6b2"
format_stamp: "Formatted at 2026-09-26 12:32:40 on dist-test-slave-3jw2"
server_key: "503340313cfd143121ef996b8777651e"
server_key_iv: "a0ae3412a33b208d7dfc1fcbbd737935"
server_key_version: "encryptionkey@0"
I20260926 12:32:40.354046 28385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:40.358691 28385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:40.361591 28385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:40.361783 28385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:40.369081 28385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41459
I20260926 12:32:40.369092 28452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41459 every 8 connection(s)
I20260926 12:32:40.370149 28385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:40.373173 28453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:40.373373 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28385
I20260926 12:32:40.373507 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:40.373800 7564 external_mini_cluster.cc:1428] Setting key 7a196a1b16d73e1b0bc5b341ad5d4f34
I20260926 12:32:40.379297 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:40.375186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46785 (local address 127.0.0.1:41459)
0926 12:32:40.375587 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:40.375605 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:40.375647 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:32:40.376438 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:40.376461 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:40.376832 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:40.377194 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:40.377931 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:40.377937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:40.377952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:40.377963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:40.378025 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:40.378050 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:40.378190 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:40.378269 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 12:32:40.378420 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260926 12:32:40.380227 28453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2: Bootstrap starting.
I20260926 12:32:40.382386 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:40.382790 28453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:40.383649 28453 log.cc:824] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:40.385800 28453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2: No bootstrap required, opened a new log
I20260926 12:32:40.386255 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:40.384716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45205 (local address 127.0.0.1:41459)
0926 12:32:40.384828 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:40.384832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:40.384995 (+ 163us) server_negotiation.cc:407] Connection header received
0926 12:32:40.385075 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:40.385078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:40.385108 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:40.385194 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:40.385892 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:40.385895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:40.385898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:40.385909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:40.385932 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:40.385944 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:40.386019 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:40.386101 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 12:32:40.386141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:32:40.389550 28453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260926 12:32:40.389775 28453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:40.389865 28453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95424705e01f48328075ead4b0e8b6b2, State: Initialized, Role: FOLLOWER
I20260926 12:32:40.390264 28453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260926 12:32:40.390414 28453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:40.390483 28453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:40.390595 28453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:40.391429 28453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260926 12:32:40.391806 28453 leader_election.cc:302] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95424705e01f48328075ead4b0e8b6b2; no voters:
I20260926 12:32:40.392045 28453 leader_election.cc:288] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:40.392200 28463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:40.392450 28463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [term 1 LEADER]: Becoming Leader. State: Replica: 95424705e01f48328075ead4b0e8b6b2, State: Running, Role: LEADER
I20260926 12:32:40.392771 28463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260926 12:32:40.393203 28453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:40.394351 28464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95424705e01f48328075ead4b0e8b6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } } }
I20260926 12:32:40.394471 28464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:40.394795 28465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95424705e01f48328075ead4b0e8b6b2. Latest consensus state: current_term: 1 leader_uuid: "95424705e01f48328075ead4b0e8b6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95424705e01f48328075ead4b0e8b6b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } } }
I20260926 12:32:40.394939 28472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:40.395479 28465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:40.398416 28472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:40.403756 28472 catalog_manager.cc:1347] Generated new cluster ID: d50512a1153444e8850d294ee6215070
I20260926 12:32:40.403817 28472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:40.413384 28472 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:40.413666 28472 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:40.422147 28472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95424705e01f48328075ead4b0e8b6b2: Generated new TSK 0
I20260926 12:32:40.422847 28472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:40.429649 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28385
2026-09-26T12:32:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:40 dist-test-slave-3jw2 krb5kdc[28484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:40 dist-test-slave-3jw2 krb5kdc[28484](info): set up 2 sockets
Sep 26 12:32:40 dist-test-slave-3jw2 krb5kdc[28484](info): commencing operation
krb5kdc: starting...
W20260926 12:32:41.468689 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:41.587584 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:41.698742 28497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:41.699083 28497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:41.699175 28497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:41.703267 28497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:41.703361 28497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:41.703400 28497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:41.703437 28497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:41.703471 28497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:41.708917 28497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:41.709980 28497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:41.710990 28497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:41.719619 28505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:41.719765 28497 server_base.cc:1034] running on GCE node
W20260926 12:32:41.719683 28502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:41.719996 28503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:41.720606 28497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:41.721721 28497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:41.722898 28497 hybrid_clock.cc:648] HybridClock initialized: now 1790425961722864 us; error 49 us; skew 500 ppm
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:41.725693 28497 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:41.726583 28497 webserver.cc:460] Webserver started at http://127.0.0.1:39453/ using document root <none> and password file <none>
I20260926 12:32:41.727113 28497 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:41.727172 28497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:41.727376 28497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:41.729388 28497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "f2247bb7fa8d495c9976600dbd46c4c4"
format_stamp: "Formatted at 2026-09-26 12:32:41 on dist-test-slave-3jw2"
server_key: "833744a88c38e3442bd962a512b0e5be"
server_key_iv: "7cc96673ddcde57b52f1e4d980a99c87"
server_key_version: "encryptionkey@0"
I20260926 12:32:41.730151 28497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "f2247bb7fa8d495c9976600dbd46c4c4"
format_stamp: "Formatted at 2026-09-26 12:32:41 on dist-test-slave-3jw2"
server_key: "833744a88c38e3442bd962a512b0e5be"
server_key_iv: "7cc96673ddcde57b52f1e4d980a99c87"
server_key_version: "encryptionkey@0"
I20260926 12:32:41.734193 28497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:32:41.736649 28512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:41.737824 28497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:41.737963 28497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "f2247bb7fa8d495c9976600dbd46c4c4"
format_stamp: "Formatted at 2026-09-26 12:32:41 on dist-test-slave-3jw2"
server_key: "833744a88c38e3442bd962a512b0e5be"
server_key_iv: "7cc96673ddcde57b52f1e4d980a99c87"
server_key_version: "encryptionkey@0"
I20260926 12:32:41.738073 28497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:41.750133 28497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:41.753672 28497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:41.753875 28497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:41.760360 28497 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45663
I20260926 12:32:41.760393 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45663 every 8 connection(s)
I20260926 12:32:41.761301 28497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:41.764009 28565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:41.764096 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28497
I20260926 12:32:41.764248 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:41.764559 7564 external_mini_cluster.cc:1428] Setting key a91d6e82a612c96e01f3488f389acf94
I20260926 12:32:41.770303 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: Bootstrap starting.
I20260926 12:32:41.772571 28565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: Neither blocks nor log segments found. Creating new log.
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:32:41.773284 28565 log.cc:824] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:41.775355 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: No bootstrap required, opened a new log
I20260926 12:32:41.778541 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:41.765979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45701 (local address 127.0.0.1:45663)
0926 12:32:41.766417 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:41.766431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:41.766465 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:41.767157 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:41.767173 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:41.767501 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:41.767850 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:41.768720 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:41.769692 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:41.770627 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:41.771090 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:41.771269 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:41.773909 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:41.773914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:41.773927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:41.773954 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:41.775673 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:41.776230 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:41.776238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:41.776244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:41.776362 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:41.776706 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:41.776709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:41.776711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:41.777090 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:41.777311 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:41.777623 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 12:32:41.777840 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260926 12:32:41.779019 28565 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260926 12:32:41.779263 28565 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:41.779361 28565 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2247bb7fa8d495c9976600dbd46c4c4, State: Initialized, Role: FOLLOWER
I20260926 12:32:41.779855 28565 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260926 12:32:41.780011 28565 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:41.780072 28565 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:41.780184 28565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:41.781291 28565 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260926 12:32:41.781678 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:41.781723 28565 leader_election.cc:302] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2247bb7fa8d495c9976600dbd46c4c4; no voters:
I20260926 12:32:41.782084 28565 leader_election.cc:288] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:41.782152 28570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:41.782339 28570 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [term 1 LEADER]: Becoming Leader. State: Replica: f2247bb7fa8d495c9976600dbd46c4c4, State: Running, Role: LEADER
I20260926 12:32:41.782685 28570 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260926 12:32:41.783313 28565 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:41.784637 28572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2247bb7fa8d495c9976600dbd46c4c4. Latest consensus state: current_term: 1 leader_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260926 12:32:41.784763 28572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:41.785249 28571 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2247bb7fa8d495c9976600dbd46c4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260926 12:32:41.785334 28571 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4 [sys.catalog]: This master's current role is: LEADER
W20260926 12:32:41.788153 28590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:41.788331 28590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:41.788440 28585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:41.789276 28585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:41.793479 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:41.785054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:44927 (local address 127.0.0.1:45663)
0926 12:32:41.785152 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:41.785159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:41.785173 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:32:41.785205 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:41.785208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:41.785250 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:41.785334 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:41.786226 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:41.786770 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:41.787847 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:41.788276 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:41.788378 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:41.790158 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:41.790162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:41.790164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:41.790188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:41.791758 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:41.792249 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:41.792253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:41.792255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:41.792331 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:41.792651 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:41.792654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:41.792656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:41.792823 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:41.792973 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:41.793226 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 12:32:41.793316 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 12:32:41.794941 28585 catalog_manager.cc:1347] Generated new cluster ID: 86224f8a24ba4c908448537d9d7ee7ec
I20260926 12:32:41.794999 28585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:41.822000 28585 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:41.823272 28585 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:41.828136 28585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2247bb7fa8d495c9976600dbd46c4c4: Generated new TSK 0
I20260926 12:32:41.828940 28585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:41.834663 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28497
W20260926 12:32:41.839954 28482 connection.cc:535] client connection to 127.0.0.1:45663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45663): Connection reset by peer (error 104)
2026-09-26T12:32:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28596](info): set up 2 sockets
Sep 26 12:32:41 dist-test-slave-3jw2 krb5kdc[28596](info): commencing operation
krb5kdc: starting...
W20260926 12:32:42.884632 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:42 dist-test-slave-3jw2 krb5kdc[28596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:43.003414 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:43.116982 28609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:43.117239 28609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:43.117282 28609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:43.121315 28609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:43.121376 28609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:43.121398 28609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:43.121417 28609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:43.121435 28609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:43.126575 28609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:43.127522 28609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:43.128543 28609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:43.136626 28614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:43.136590 28615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:43.136806 28609 server_base.cc:1034] running on GCE node
W20260926 12:32:43.136924 28617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:43.137486 28609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:43.138454 28609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:43.139647 28609 hybrid_clock.cc:648] HybridClock initialized: now 1790425963139629 us; error 33 us; skew 500 ppm
Sep 26 12:32:43 dist-test-slave-3jw2 krb5kdc[28596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:43.142378 28609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:43.143129 28609 webserver.cc:460] Webserver started at http://127.0.0.1:42043/ using document root <none> and password file <none>
I20260926 12:32:43.143603 28609 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:43.143657 28609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:43.143815 28609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:43.145793 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "0ac08af3ac9c4536a5f3587ca57862c6"
format_stamp: "Formatted at 2026-09-26 12:32:43 on dist-test-slave-3jw2"
server_key: "6606bc8b47f0612a623b42ce9dee322e"
server_key_iv: "8fed485b192998726da62391e862d0a5"
server_key_version: "encryptionkey@0"
I20260926 12:32:43.146258 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "0ac08af3ac9c4536a5f3587ca57862c6"
format_stamp: "Formatted at 2026-09-26 12:32:43 on dist-test-slave-3jw2"
server_key: "6606bc8b47f0612a623b42ce9dee322e"
server_key_iv: "8fed485b192998726da62391e862d0a5"
server_key_version: "encryptionkey@0"
I20260926 12:32:43.149818 28609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:32:43.152022 28624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:43.153254 28609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:43.153354 28609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "0ac08af3ac9c4536a5f3587ca57862c6"
format_stamp: "Formatted at 2026-09-26 12:32:43 on dist-test-slave-3jw2"
server_key: "6606bc8b47f0612a623b42ce9dee322e"
server_key_iv: "8fed485b192998726da62391e862d0a5"
server_key_version: "encryptionkey@0"
I20260926 12:32:43.153437 28609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:43.167689 28609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:43.170855 28609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:43.171001 28609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:43.178071 28609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37029
I20260926 12:32:43.178155 28676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37029 every 8 connection(s)
I20260926 12:32:43.178997 28609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:43.179286 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28609
I20260926 12:32:43.179436 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:43.179775 7564 external_mini_cluster.cc:1428] Setting key 4c2c96a16dda4b00481168e4b7c41804
I20260926 12:32:43.182157 28677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:43.188520 28677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6: Bootstrap starting.
I20260926 12:32:43.188692 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:43.181138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:38615 (local address 127.0.0.1:37029)
0926 12:32:43.181558 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:43.181574 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:43.181609 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:32:43.182246 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:43.182270 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:43.182608 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:43.182878 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:43.183210 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:43.184292 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:43.185525 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:43.185984 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:43.186094 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:43.186976 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:43.186981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:43.186992 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:43.187346 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:43.187394 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:43.187414 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:43.187511 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:43.187752 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 12:32:43.187893 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260926 12:32:43.190763 28677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:43.191505 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:43.191643 28677 log.cc:824] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:43.193830 28677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6: No bootstrap required, opened a new log
I20260926 12:32:43.197603 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:43.193684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:33363 (local address 127.0.0.1:37029)
0926 12:32:43.193812 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:43.193816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:43.193962 (+ 146us) server_negotiation.cc:407] Connection header received
0926 12:32:43.194097 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:43.194100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:43.194144 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:43.194241 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:43.194463 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:43.195186 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:43.195964 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:43.196324 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:43.196430 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:43.197197 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:43.197200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:43.197203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:43.197215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:43.197240 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:43.197256 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:43.197350 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:43.197446 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:32:43.197478 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:32:43.198045 28677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260926 12:32:43.198247 28677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:43.198316 28677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac08af3ac9c4536a5f3587ca57862c6, State: Initialized, Role: FOLLOWER
I20260926 12:32:43.198743 28677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260926 12:32:43.198895 28677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:43.198959 28677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:43.199061 28677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:43.199925 28677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260926 12:32:43.200312 28677 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac08af3ac9c4536a5f3587ca57862c6; no voters:
I20260926 12:32:43.200614 28677 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:43.200680 28687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:43.200938 28687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac08af3ac9c4536a5f3587ca57862c6, State: Running, Role: LEADER
I20260926 12:32:43.201268 28687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260926 12:32:43.201820 28677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:43.202948 28688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } } }
I20260926 12:32:43.202922 28689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac08af3ac9c4536a5f3587ca57862c6. Latest consensus state: current_term: 1 leader_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac08af3ac9c4536a5f3587ca57862c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } } }
I20260926 12:32:43.203094 28688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:43.203099 28689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:43.203522 28696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:43.206697 28696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:43.211774 28696 catalog_manager.cc:1347] Generated new cluster ID: 11783d02d4bb4d1d86ae49e961c55575
I20260926 12:32:43.211850 28696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:43.237440 28696 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:43.238689 28696 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:43.246249 28696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac08af3ac9c4536a5f3587ca57862c6: Generated new TSK 0
I20260926 12:32:43.246978 28696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:43.254428 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28609
W20260926 12:32:43.259769 28594 connection.cc:535] client connection to 127.0.0.1:37029 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37029): Connection reset by peer (error 104)
2026-09-26T12:32:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:43 dist-test-slave-3jw2 krb5kdc[28708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:43 dist-test-slave-3jw2 krb5kdc[28708](info): set up 2 sockets
Sep 26 12:32:43 dist-test-slave-3jw2 krb5kdc[28708](info): commencing operation
krb5kdc: starting...
W20260926 12:32:44.280632 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:44 dist-test-slave-3jw2 krb5kdc[28708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:44.402797 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:44.512426 28721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:44.512720 28721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:44.512810 28721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:44.516883 28721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:44.516969 28721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:44.517006 28721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:44.517068 28721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:44.517112 28721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:44.522699 28721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:44.523841 28721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:44.524982 28721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:44.533854 28729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:44.533819 28726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:44.533836 28727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:44.533924 28721 server_base.cc:1034] running on GCE node
I20260926 12:32:44.534647 28721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:44.535817 28721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:44.537000 28721 hybrid_clock.cc:648] HybridClock initialized: now 1790425964536997 us; error 37 us; skew 500 ppm
Sep 26 12:32:44 dist-test-slave-3jw2 krb5kdc[28708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:44.547008 28721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:44.548009 28721 webserver.cc:460] Webserver started at http://127.0.0.1:33063/ using document root <none> and password file <none>
I20260926 12:32:44.548642 28721 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:44.548740 28721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:44.548977 28721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:44.551059 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "eccc00dc8eda4ff088ce5bbd5f864a36"
format_stamp: "Formatted at 2026-09-26 12:32:44 on dist-test-slave-3jw2"
server_key: "8d4b804940d6e8d68fabc43eaf5a3922"
server_key_iv: "83bd666b7e15659c54d5e73045dc629a"
server_key_version: "encryptionkey@0"
I20260926 12:32:44.551641 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "eccc00dc8eda4ff088ce5bbd5f864a36"
format_stamp: "Formatted at 2026-09-26 12:32:44 on dist-test-slave-3jw2"
server_key: "8d4b804940d6e8d68fabc43eaf5a3922"
server_key_iv: "83bd666b7e15659c54d5e73045dc629a"
server_key_version: "encryptionkey@0"
I20260926 12:32:44.555846 28721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 12:32:44.558413 28736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:44.559818 28721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:32:44.559971 28721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "eccc00dc8eda4ff088ce5bbd5f864a36"
format_stamp: "Formatted at 2026-09-26 12:32:44 on dist-test-slave-3jw2"
server_key: "8d4b804940d6e8d68fabc43eaf5a3922"
server_key_iv: "83bd666b7e15659c54d5e73045dc629a"
server_key_version: "encryptionkey@0"
I20260926 12:32:44.560098 28721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:44.564749 28721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:44.567942 28721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:44.568143 28721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:44.575446 28721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37825
I20260926 12:32:44.575467 28788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37825 every 8 connection(s)
I20260926 12:32:44.576454 28721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:44.578868 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28721
I20260926 12:32:44.579025 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:44.579360 7564 external_mini_cluster.cc:1428] Setting key a761aa636afcc2fca581ee1485701308
I20260926 12:32:44.579455 28789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:44.584780 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:44.580814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:46187 (local address 127.0.0.1:37825)
0926 12:32:44.581252 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:44.581266 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:44.581300 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:44.582144 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:44.582160 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:44.582441 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:44.582774 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:44.583534 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:44.583539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:44.583550 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:44.583561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:44.583612 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:44.583630 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:44.583812 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:44.583897 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:32:44.584054 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260926 12:32:44.586076 28789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36: Bootstrap starting.
I20260926 12:32:44.587549 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:44.588513 28789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:44.589287 28789 log.cc:824] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:44.591353 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:44.589661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:43381 (local address 127.0.0.1:37825)
0926 12:32:44.589817 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:44.589821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:44.590045 (+ 224us) server_negotiation.cc:407] Connection header received
0926 12:32:44.590156 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:44.590158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:44.590185 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:44.590253 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:44.590923 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:44.590926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:44.590928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:44.590937 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:44.590961 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:44.590970 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:44.591038 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:44.591163 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 12:32:44.591197 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 12:32:44.591692 28789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36: No bootstrap required, opened a new log
I20260926 12:32:44.595211 28789 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } }
I20260926 12:32:44.595427 28789 raft_consensus.cc:374] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:44.595461 28789 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccc00dc8eda4ff088ce5bbd5f864a36, State: Initialized, Role: FOLLOWER
I20260926 12:32:44.595829 28789 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } }
I20260926 12:32:44.595940 28789 raft_consensus.cc:388] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:44.595984 28789 raft_consensus.cc:482] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:44.596063 28789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:44.597210 28789 raft_consensus.cc:504] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } }
I20260926 12:32:44.597538 28789 leader_election.cc:302] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eccc00dc8eda4ff088ce5bbd5f864a36; no voters:
I20260926 12:32:44.597872 28789 leader_election.cc:288] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:44.597998 28799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:44.598281 28799 raft_consensus.cc:686] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [term 1 LEADER]: Becoming Leader. State: Replica: eccc00dc8eda4ff088ce5bbd5f864a36, State: Running, Role: LEADER
I20260926 12:32:44.598626 28799 consensus_queue.cc:237] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } }
I20260926 12:32:44.599234 28789 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:44.600538 28800 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } } }
I20260926 12:32:44.600512 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eccc00dc8eda4ff088ce5bbd5f864a36. Latest consensus state: current_term: 1 leader_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccc00dc8eda4ff088ce5bbd5f864a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37825 } } }
I20260926 12:32:44.600666 28800 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:44.600674 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:44.601795 28808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:44.604493 28808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:44.610102 28808 catalog_manager.cc:1347] Generated new cluster ID: ef174c80677f49cbbbaf4952b11a8bbe
I20260926 12:32:44.610175 28808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:44.618366 28808 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:44.618693 28808 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:44.630551 28808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eccc00dc8eda4ff088ce5bbd5f864a36: Generated new TSK 0
I20260926 12:32:44.631394 28808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:44.641378 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28721
2026-09-26T12:32:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:44 dist-test-slave-3jw2 krb5kdc[28820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:44 dist-test-slave-3jw2 krb5kdc[28820](info): set up 2 sockets
Sep 26 12:32:44 dist-test-slave-3jw2 krb5kdc[28820](info): commencing operation
krb5kdc: starting...
W20260926 12:32:45.672660 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:45 dist-test-slave-3jw2 krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:45.791183 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:45.901898 28833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:45.902148 28833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:45.902187 28833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:45.906280 28833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:45.906335 28833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:45.906354 28833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:45.906371 28833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:45.906385 28833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:45.911497 28833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:45.912549 28833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:45.913555 28833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:45.921707 28839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:45.921720 28841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:45.921708 28838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:45.921998 28833 server_base.cc:1034] running on GCE node
I20260926 12:32:45.922533 28833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:45.923530 28833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:45.924715 28833 hybrid_clock.cc:648] HybridClock initialized: now 1790425965924700 us; error 35 us; skew 500 ppm
Sep 26 12:32:45 dist-test-slave-3jw2 krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:45.927306 28833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:45.928117 28833 webserver.cc:460] Webserver started at http://127.0.0.1:39401/ using document root <none> and password file <none>
I20260926 12:32:45.928679 28833 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:45.928766 28833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:45.928987 28833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:45.930979 28833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "cb93dcdaad6044b0a7c792fec6961601"
format_stamp: "Formatted at 2026-09-26 12:32:45 on dist-test-slave-3jw2"
server_key: "1ca5a74da79d1df41f7e4a1e51982a16"
server_key_iv: "c075b88ee4a678680b02efa783fb61e6"
server_key_version: "encryptionkey@0"
I20260926 12:32:45.931535 28833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "cb93dcdaad6044b0a7c792fec6961601"
format_stamp: "Formatted at 2026-09-26 12:32:45 on dist-test-slave-3jw2"
server_key: "1ca5a74da79d1df41f7e4a1e51982a16"
server_key_iv: "c075b88ee4a678680b02efa783fb61e6"
server_key_version: "encryptionkey@0"
I20260926 12:32:45.935191 28833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:32:45.937615 28848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:45.938834 28833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:45.938971 28833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "cb93dcdaad6044b0a7c792fec6961601"
format_stamp: "Formatted at 2026-09-26 12:32:45 on dist-test-slave-3jw2"
server_key: "1ca5a74da79d1df41f7e4a1e51982a16"
server_key_iv: "c075b88ee4a678680b02efa783fb61e6"
server_key_version: "encryptionkey@0"
I20260926 12:32:45.939080 28833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:45.962437 28833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:45.965401 28833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:45.965605 28833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:45.972566 28833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34437
I20260926 12:32:45.972590 28900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34437 every 8 connection(s)
I20260926 12:32:45.973495 28833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:45.976226 28901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:45.977190 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28833
I20260926 12:32:45.977315 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:45.977602 7564 external_mini_cluster.cc:1428] Setting key 368f8d678db737de355460347bb2003c
I20260926 12:32:45.982963 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601: Bootstrap starting.
I20260926 12:32:45.985690 28901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:45.986589 28901 log.cc:824] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:32:45 dist-test-slave-3jw2 krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:32:45.988755 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601: No bootstrap required, opened a new log
I20260926 12:32:45.992100 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:45.979045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:44715 (local address 127.0.0.1:34437)
0926 12:32:45.979450 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:45.979474 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:45.979508 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:45.980119 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:45.980135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:45.980541 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:45.980815 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:45.981795 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:45.982744 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:45.983889 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:45.984442 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:45.984565 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:45.987488 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:45.987493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:45.987505 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:45.987530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:45.989295 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:45.989824 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:45.989828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:45.989833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:45.989934 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:45.990269 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:45.990272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:45.990274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:45.990628 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:45.990850 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:45.991173 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 12:32:45.991394 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260926 12:32:45.992383 28901 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } }
I20260926 12:32:45.992666 28901 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:45.992729 28901 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb93dcdaad6044b0a7c792fec6961601, State: Initialized, Role: FOLLOWER
I20260926 12:32:45.993203 28901 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } }
I20260926 12:32:45.993387 28901 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:45.993491 28901 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:45.993618 28901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:45.994704 28901 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } }
I20260926 12:32:45.995031 28901 leader_election.cc:302] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb93dcdaad6044b0a7c792fec6961601; no voters:
I20260926 12:32:45.995363 28901 leader_election.cc:288] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:45.995390 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:45.995456 28906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:45.995713 28906 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [term 1 LEADER]: Becoming Leader. State: Replica: cb93dcdaad6044b0a7c792fec6961601, State: Running, Role: LEADER
I20260926 12:32:45.996030 28906 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } }
I20260926 12:32:45.996765 28901 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:45.997716 28908 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb93dcdaad6044b0a7c792fec6961601. Latest consensus state: current_term: 1 leader_uuid: "cb93dcdaad6044b0a7c792fec6961601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } } }
I20260926 12:32:45.997740 28907 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb93dcdaad6044b0a7c792fec6961601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb93dcdaad6044b0a7c792fec6961601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34437 } } }
I20260926 12:32:45.997848 28908 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:45.997848 28907 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:45.998239 28919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:46.001106 28919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:46.007102 28919 catalog_manager.cc:1347] Generated new cluster ID: 47d1cee9fec44d41a6f038ac5c5966b9
I20260926 12:32:46.007203 28919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:46.009419 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:45.999247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:51649 (local address 127.0.0.1:34437)
0926 12:32:45.999353 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:45.999357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:46.000460 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 12:32:46.000591 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:46.000594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:46.000641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:46.000728 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:46.002077 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:46.002811 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:46.003632 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:46.003859 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:46.003963 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:46.005566 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:46.005571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:46.005574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:46.005600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:46.007726 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:46.008202 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:46.008206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:46.008208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:46.008279 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:46.008598 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:46.008601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:46.008603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:46.008761 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:46.008928 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:46.009182 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 12:32:46.009270 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 12:32:46.020833 28919 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:46.021704 28919 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:46.029443 28919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb93dcdaad6044b0a7c792fec6961601: Generated new TSK 0
I20260926 12:32:46.030072 28919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:46.037827 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28833
W20260926 12:32:46.043161 28818 connection.cc:535] client connection to 127.0.0.1:34437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34437): Connection reset by peer (error 104)
2026-09-26T12:32:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:46 dist-test-slave-3jw2 krb5kdc[28932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:46 dist-test-slave-3jw2 krb5kdc[28932](info): set up 2 sockets
Sep 26 12:32:46 dist-test-slave-3jw2 krb5kdc[28932](info): commencing operation
krb5kdc: starting...
W20260926 12:32:47.068657 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:47 dist-test-slave-3jw2 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:47.193156 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:47.299957 28945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:47.300278 28945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:47.300362 28945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:47.304339 28945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:47.304428 28945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:47.304467 28945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:47.304528 28945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:47.304569 28945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:47.309604 28945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:47.310616 28945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:47.311602 28945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:47.319756 28951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:47.319741 28950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:47.319758 28953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:47.320101 28945 server_base.cc:1034] running on GCE node
I20260926 12:32:47.320664 28945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:47.321740 28945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:47.322928 28945 hybrid_clock.cc:648] HybridClock initialized: now 1790425967322921 us; error 30 us; skew 500 ppm
Sep 26 12:32:47 dist-test-slave-3jw2 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:47.325738 28945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:47.326617 28945 webserver.cc:460] Webserver started at http://127.0.0.1:34687/ using document root <none> and password file <none>
I20260926 12:32:47.327153 28945 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:47.327247 28945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:47.327474 28945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:47.329547 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "13a4a2b62c624011a60712944c278653"
format_stamp: "Formatted at 2026-09-26 12:32:47 on dist-test-slave-3jw2"
server_key: "6300b80802c5c3c34851c0dee378cb04"
server_key_iv: "bd37eea6e349df2a51da537da1cb69e7"
server_key_version: "encryptionkey@0"
I20260926 12:32:47.330125 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "13a4a2b62c624011a60712944c278653"
format_stamp: "Formatted at 2026-09-26 12:32:47 on dist-test-slave-3jw2"
server_key: "6300b80802c5c3c34851c0dee378cb04"
server_key_iv: "bd37eea6e349df2a51da537da1cb69e7"
server_key_version: "encryptionkey@0"
I20260926 12:32:47.333810 28945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:32:47.336050 28960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:47.337270 28945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:47.337406 28945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "13a4a2b62c624011a60712944c278653"
format_stamp: "Formatted at 2026-09-26 12:32:47 on dist-test-slave-3jw2"
server_key: "6300b80802c5c3c34851c0dee378cb04"
server_key_iv: "bd37eea6e349df2a51da537da1cb69e7"
server_key_version: "encryptionkey@0"
I20260926 12:32:47.337513 28945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:47.355427 28945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:47.358821 28945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:47.359031 28945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:47.366444 28945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37213
I20260926 12:32:47.366472 29012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37213 every 8 connection(s)
I20260926 12:32:47.367424 28945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:47.369062 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 28945
I20260926 12:32:47.369181 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:47.369498 7564 external_mini_cluster.cc:1428] Setting key 492a922228efe9e9627beaf4c952e12e
I20260926 12:32:47.370599 29013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:47.376847 29013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653: Bootstrap starting.
I20260926 12:32:47.378677 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:47.370922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:54533 (local address 127.0.0.1:37213)
0926 12:32:47.371344 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:47.371370 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:47.371404 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:47.372094 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:47.372110 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:47.372518 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:47.372801 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:47.373153 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:47.374164 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:47.375377 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:47.375904 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:47.376038 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:47.376933 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:47.376937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:47.376949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:47.377363 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:47.377414 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:47.377436 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:47.377536 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:47.377739 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 12:32:47.377893 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":143,"threads_started":1}
I20260926 12:32:47.379264 29013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:47.380048 29013 log.cc:824] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:47.381669 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:47.382333 29013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653: No bootstrap required, opened a new log
I20260926 12:32:47.386087 29013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260926 12:32:47.386322 29013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:47.386405 29013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a4a2b62c624011a60712944c278653, State: Initialized, Role: FOLLOWER
I20260926 12:32:47.386889 29013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260926 12:32:47.387039 29013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:47.387126 29013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:47.387267 29013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:47.387537 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:47.383656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:33727 (local address 127.0.0.1:37213)
0926 12:32:47.383775 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:47.383779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:47.383892 (+ 113us) server_negotiation.cc:407] Connection header received
0926 12:32:47.384040 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:47.384043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:47.384090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:47.384194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:47.384451 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:47.384957 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:47.385779 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:47.386010 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:47.386127 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:47.387049 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:47.387052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:47.387055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:47.387067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:47.387099 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:47.387110 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:47.387230 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:47.387353 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 12:32:47.387390 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:47.388351 29013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260926 12:32:47.388689 29013 leader_election.cc:302] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a4a2b62c624011a60712944c278653; no voters:
I20260926 12:32:47.388993 29013 leader_election.cc:288] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:47.389106 29023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:47.389333 29023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [term 1 LEADER]: Becoming Leader. State: Replica: 13a4a2b62c624011a60712944c278653, State: Running, Role: LEADER
I20260926 12:32:47.389668 29023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260926 12:32:47.390213 29013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:47.391407 29025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a4a2b62c624011a60712944c278653. Latest consensus state: current_term: 1 leader_uuid: "13a4a2b62c624011a60712944c278653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } } }
I20260926 12:32:47.391412 29024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13a4a2b62c624011a60712944c278653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a4a2b62c624011a60712944c278653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } } }
I20260926 12:32:47.391609 29024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:47.391607 29025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:47.392038 29033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:47.394887 29033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:47.400377 29033 catalog_manager.cc:1347] Generated new cluster ID: af9a534af9154162b73ae4d670966f54
I20260926 12:32:47.400451 29033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:47.422282 29033 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:47.423208 29033 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:47.432085 29033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13a4a2b62c624011a60712944c278653: Generated new TSK 0
I20260926 12:32:47.432932 29033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:47.445895 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 28945
W20260926 12:32:47.451288 28930 connection.cc:535] client connection to 127.0.0.1:37213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37213): Connection reset by peer (error 104)
2026-09-26T12:32:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:47 dist-test-slave-3jw2 krb5kdc[29044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:47 dist-test-slave-3jw2 krb5kdc[29044](info): set up 2 sockets
Sep 26 12:32:47 dist-test-slave-3jw2 krb5kdc[29044](info): commencing operation
krb5kdc: starting...
W20260926 12:32:48.564608 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:48 dist-test-slave-3jw2 krb5kdc[29044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:32:48.684561 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:48.796458 29057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:48.796700 29057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:48.796741 29057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:48.800777 29057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:48.800835 29057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:48.800854 29057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:48.800916 29057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:48.800951 29057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:48.806078 29057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:48.807134 29057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:48.808239 29057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:48.816476 29063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:48.816629 29057 server_base.cc:1034] running on GCE node
W20260926 12:32:48.816514 29062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:48.816479 29065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:48.817282 29057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:48.818356 29057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:48.819525 29057 hybrid_clock.cc:648] HybridClock initialized: now 1790425968819505 us; error 28 us; skew 500 ppm
Sep 26 12:32:48 dist-test-slave-3jw2 krb5kdc[29044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:48.822388 29057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:32:48.823310 29057 webserver.cc:460] Webserver started at http://127.0.0.1:40845/ using document root <none> and password file <none>
I20260926 12:32:48.823849 29057 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:48.823940 29057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:48.824218 29057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:48.826229 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "6a695d1a040e4df18f83e69c215e1e26"
format_stamp: "Formatted at 2026-09-26 12:32:48 on dist-test-slave-3jw2"
server_key: "249dc8d9361542f94efc64a9b311cd91"
server_key_iv: "c3426641746917a3dce32f7fcac0e04f"
server_key_version: "encryptionkey@0"
I20260926 12:32:48.826797 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "6a695d1a040e4df18f83e69c215e1e26"
format_stamp: "Formatted at 2026-09-26 12:32:48 on dist-test-slave-3jw2"
server_key: "249dc8d9361542f94efc64a9b311cd91"
server_key_iv: "c3426641746917a3dce32f7fcac0e04f"
server_key_version: "encryptionkey@0"
I20260926 12:32:48.830651 29057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 12:32:48.833068 29072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:48.834352 29057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:48.834502 29057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "6a695d1a040e4df18f83e69c215e1e26"
format_stamp: "Formatted at 2026-09-26 12:32:48 on dist-test-slave-3jw2"
server_key: "249dc8d9361542f94efc64a9b311cd91"
server_key_iv: "c3426641746917a3dce32f7fcac0e04f"
server_key_version: "encryptionkey@0"
I20260926 12:32:48.834617 29057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:48.839051 29057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:48.841948 29057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:48.842178 29057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:48.848973 29057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35375
I20260926 12:32:48.848999 29124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35375 every 8 connection(s)
I20260926 12:32:48.850252 29057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:48.853670 29125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:48.859835 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: Bootstrap starting.
I20260926 12:32:48.860227 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29057
I20260926 12:32:48.860360 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:48.860641 7564 external_mini_cluster.cc:1428] Setting key 0eb7e2f31c3f68d364d64e83993be7bb
I20260926 12:32:48.862653 29125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:48.863442 29125 log.cc:824] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:48.865698 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: No bootstrap required, opened a new log
I20260926 12:32:48.866101 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:48.861950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:33389 (local address 127.0.0.1:35375)
0926 12:32:48.862429 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:48.862443 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:48.862477 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:48.863241 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:48.863259 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:48.863565 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:48.863951 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:48.864756 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:48.864761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:48.864775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:48.864786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:48.864844 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:48.864870 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:48.864982 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:48.865071 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 12:32:48.865220 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":134,"threads_started":1}
I20260926 12:32:48.868860 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:48.869797 29125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260926 12:32:48.870033 29125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:48.870117 29125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a695d1a040e4df18f83e69c215e1e26, State: Initialized, Role: FOLLOWER
I20260926 12:32:48.870576 29125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260926 12:32:48.870726 29125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:48.870814 29125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:48.870949 29125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:48.871959 29125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260926 12:32:48.872390 29125 leader_election.cc:302] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a695d1a040e4df18f83e69c215e1e26; no voters:
I20260926 12:32:48.872512 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:48.870879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:49259 (local address 127.0.0.1:35375)
0926 12:32:48.870991 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:48.870995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:48.871178 (+ 183us) server_negotiation.cc:407] Connection header received
0926 12:32:48.871248 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:48.871251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:48.871287 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:48.871367 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:48.872053 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:48.872058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:48.872062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:48.872073 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:48.872099 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:48.872111 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:48.872213 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:48.872363 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 12:32:48.872404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:32:48.872742 29125 leader_election.cc:288] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:48.873961 29125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:48.875885 29136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:48.876475 29136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [term 1 LEADER]: Becoming Leader. State: Replica: 6a695d1a040e4df18f83e69c215e1e26, State: Running, Role: LEADER
I20260926 12:32:48.876758 29136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260926 12:32:48.877925 29136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a695d1a040e4df18f83e69c215e1e26. Latest consensus state: current_term: 1 leader_uuid: "6a695d1a040e4df18f83e69c215e1e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } } }
I20260926 12:32:48.878089 29136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:48.878746 29135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a695d1a040e4df18f83e69c215e1e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a695d1a040e4df18f83e69c215e1e26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } } }
I20260926 12:32:48.878880 29135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26 [sys.catalog]: This master's current role is: LEADER
W20260926 12:32:48.881686 29147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:48.881765 29147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:48.881842 29150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:48.882522 29150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:48.887699 29150 catalog_manager.cc:1347] Generated new cluster ID: 670da5f29b384317b4ce09b5595614c9
I20260926 12:32:48.887763 29150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:48.917900 29150 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:48.918259 29150 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:48.923210 29150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a695d1a040e4df18f83e69c215e1e26: Generated new TSK 0
I20260926 12:32:48.923890 29150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:48.929557 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29057
2026-09-26T12:32:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:48 dist-test-slave-3jw2 krb5kdc[29156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:48 dist-test-slave-3jw2 krb5kdc[29156](info): set up 2 sockets
Sep 26 12:32:48 dist-test-slave-3jw2 krb5kdc[29156](info): commencing operation
krb5kdc: starting...
W20260926 12:32:49.960817 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:50.081483 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:34685
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:50.190116 29169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:50.190433 29169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:50.190505 29169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:50.194559 29169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:50.194651 29169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:50.194691 29169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:50.194728 29169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:50.194768 29169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:50.200088 29169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:34685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:50.201171 29169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:50.202183 29169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:50.210872 29174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:50.210850 29175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:50.210851 29177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:50.211112 29169 server_base.cc:1034] running on GCE node
I20260926 12:32:50.211699 29169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:50.212844 29169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:50.214005 29169 hybrid_clock.cc:648] HybridClock initialized: now 1790425970213988 us; error 31 us; skew 500 ppm
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425970, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:50.223068 29169 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:50.224045 29169 webserver.cc:460] Webserver started at http://127.7.99.62:36557/ using document root <none> and password file <none>
I20260926 12:32:50.224684 29169 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:50.224786 29169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:50.225057 29169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:50.227111 29169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "def6edff8feb445f932fe0c77e03f25a"
format_stamp: "Formatted at 2026-09-26 12:32:50 on dist-test-slave-3jw2"
server_key: "66f3468a42323ee3933e8cb1aa24b0e6"
server_key_iv: "170c48d19d5b896566683697f3260719"
server_key_version: "encryptionkey@0"
I20260926 12:32:50.227715 29169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "def6edff8feb445f932fe0c77e03f25a"
format_stamp: "Formatted at 2026-09-26 12:32:50 on dist-test-slave-3jw2"
server_key: "66f3468a42323ee3933e8cb1aa24b0e6"
server_key_iv: "170c48d19d5b896566683697f3260719"
server_key_version: "encryptionkey@0"
I20260926 12:32:50.231477 29169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:32:50.233932 29184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:50.235183 29169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:50.235328 29169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "def6edff8feb445f932fe0c77e03f25a"
format_stamp: "Formatted at 2026-09-26 12:32:50 on dist-test-slave-3jw2"
server_key: "66f3468a42323ee3933e8cb1aa24b0e6"
server_key_iv: "170c48d19d5b896566683697f3260719"
server_key_version: "encryptionkey@0"
I20260926 12:32:50.235440 29169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:50.254105 29169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:50.257479 29169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:50.257705 29169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:50.265328 29169 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:34685
I20260926 12:32:50.265352 29236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:34685 every 8 connection(s)
I20260926 12:32:50.266283 29169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:50.267062 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29169
I20260926 12:32:50.267172 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:50.267460 7564 external_mini_cluster.cc:1428] Setting key 4cd96ca0681814c9b914a69b800e9acc
I20260926 12:32:50.269452 29237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:50.274799 29237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a: Bootstrap starting.
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:50.277354 29237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:50.278038 29237 log.cc:824] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:50.280244 29237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a: No bootstrap required, opened a new log
I20260926 12:32:50.282173 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:50.268893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49779 (local address 127.7.99.62:34685)
0926 12:32:50.269323 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:50.269341 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:50.269375 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:50.270087 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:50.270104 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:50.270440 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:50.270777 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:50.271585 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:50.272663 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:50.273548 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:50.274020 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:50.274196 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:50.277022 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:50.277027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:50.277039 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:50.277106 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:50.279108 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:50.279584 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:50.279589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:50.279595 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:50.279736 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:50.280039 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:50.280042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:50.280044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:50.280564 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:50.280775 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:50.281032 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 12:32:50.281248 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
I20260926 12:32:50.284474 29237 raft_consensus.cc:348] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } }
I20260926 12:32:50.284796 29237 raft_consensus.cc:374] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:50.284876 29237 raft_consensus.cc:729] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def6edff8feb445f932fe0c77e03f25a, State: Initialized, Role: FOLLOWER
I20260926 12:32:50.285241 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:50.285355 29237 consensus_queue.cc:260] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } }
I20260926 12:32:50.285514 29237 raft_consensus.cc:388] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:50.285578 29237 raft_consensus.cc:482] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:50.285668 29237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:50.286551 29237 raft_consensus.cc:504] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } }
I20260926 12:32:50.286880 29237 leader_election.cc:302] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def6edff8feb445f932fe0c77e03f25a; no voters:
I20260926 12:32:50.287199 29237 leader_election.cc:288] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:50.287451 29246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:50.287672 29246 raft_consensus.cc:686] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [term 1 LEADER]: Becoming Leader. State: Replica: def6edff8feb445f932fe0c77e03f25a, State: Running, Role: LEADER
I20260926 12:32:50.288033 29246 consensus_queue.cc:237] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } }
I20260926 12:32:50.288466 29237 sys_catalog.cc:564] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:50.289289 29246 sys_catalog.cc:455] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [sys.catalog]: SysCatalogTable state changed. Reason: New leader def6edff8feb445f932fe0c77e03f25a. Latest consensus state: current_term: 1 leader_uuid: "def6edff8feb445f932fe0c77e03f25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } } }
I20260926 12:32:50.289477 29246 sys_catalog.cc:458] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:50.289732 29248 sys_catalog.cc:455] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def6edff8feb445f932fe0c77e03f25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def6edff8feb445f932fe0c77e03f25a" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 34685 } } }
I20260926 12:32:50.289888 29248 sys_catalog.cc:458] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:50.289934 29256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:50.293022 29256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:50.297196 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:50.287292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56617 (local address 127.7.99.62:34685)
0926 12:32:50.287413 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:50.287418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:50.287873 (+ 455us) server_negotiation.cc:407] Connection header received
0926 12:32:50.287914 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:50.287917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:50.287959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:50.288040 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:50.288842 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:50.289648 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:50.290958 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:50.291321 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:50.291423 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:50.293151 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:50.293155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:50.293157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:50.293183 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:50.294829 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:50.295484 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:50.295488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:50.295489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:50.295573 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:50.295953 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:50.295957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:50.295958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:50.296197 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:50.296333 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:50.296724 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 12:32:50.296824 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 12:32:50.298635 29256 catalog_manager.cc:1347] Generated new cluster ID: 6208169fc6d54e18818895aa626951b4
I20260926 12:32:50.298702 29256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:50.312258 29256 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:50.313122 29256 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:50.320379 29256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P def6edff8feb445f932fe0c77e03f25a: Generated new TSK 0
I20260926 12:32:50.321077 29256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:50.330947 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29169
W20260926 12:32:50.336385 29154 connection.cc:535] client connection to 127.7.99.62:34685 recv error: Network error: failed to read from TLS socket (remote: 127.7.99.62:34685): Connection reset by peer (error 104)
2026-09-26T12:32:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29268](info): set up 2 sockets
Sep 26 12:32:50 dist-test-slave-3jw2 krb5kdc[29268](info): commencing operation
krb5kdc: starting...
W20260926 12:32:51.380648 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:51 dist-test-slave-3jw2 krb5kdc[29268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:51.503279 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41331
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:51.613411 29281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:51.613651 29281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:51.613716 29281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:51.617688 29281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:51.617738 29281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:51.617758 29281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:51.617774 29281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:51.617826 29281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:51.622944 29281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:41331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:51.623909 29281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:51.624957 29281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:51.633358 29286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:51.633376 29289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:51.633315 29287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:51.633630 29281 server_base.cc:1034] running on GCE node
I20260926 12:32:51.634192 29281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:51.635269 29281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:51.636480 29281 hybrid_clock.cc:648] HybridClock initialized: now 1790425971636451 us; error 40 us; skew 500 ppm
Sep 26 12:32:51 dist-test-slave-3jw2 krb5kdc[29268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425971, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:51.639266 29281 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:51.640144 29281 webserver.cc:460] Webserver started at http://127.7.99.62:36475/ using document root <none> and password file <none>
I20260926 12:32:51.640717 29281 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:51.640775 29281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:51.640995 29281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:51.642975 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "6f527181d1cf419fbac0fa67518980bc"
format_stamp: "Formatted at 2026-09-26 12:32:51 on dist-test-slave-3jw2"
server_key: "fab996ccba4dcee59ae429403288bd51"
server_key_iv: "6ffb7e473dc33fdce263dffe95a808bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:51.643533 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "6f527181d1cf419fbac0fa67518980bc"
format_stamp: "Formatted at 2026-09-26 12:32:51 on dist-test-slave-3jw2"
server_key: "fab996ccba4dcee59ae429403288bd51"
server_key_iv: "6ffb7e473dc33fdce263dffe95a808bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:51.647264 29281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:32:51.649719 29296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:51.650920 29281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:51.651053 29281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "6f527181d1cf419fbac0fa67518980bc"
format_stamp: "Formatted at 2026-09-26 12:32:51 on dist-test-slave-3jw2"
server_key: "fab996ccba4dcee59ae429403288bd51"
server_key_iv: "6ffb7e473dc33fdce263dffe95a808bc"
server_key_version: "encryptionkey@0"
I20260926 12:32:51.651151 29281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:51.665503 29281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:51.668717 29281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:51.668913 29281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:51.675627 29281 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:41331
I20260926 12:32:51.675635 29348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:41331 every 8 connection(s)
I20260926 12:32:51.676621 29281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:51.679270 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29281
I20260926 12:32:51.679404 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:51.679399 29349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:51.679728 7564 external_mini_cluster.cc:1428] Setting key d093bce69067e4cfb0ce036a18a2977b
I20260926 12:32:51.685775 29349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: Bootstrap starting.
I20260926 12:32:51.688014 29349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:51.688495 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:51.681191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50277 (local address 127.7.99.62:41331)
0926 12:32:51.681625 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:51.681639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:51.681672 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:32:51.682268 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:51.682283 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:51.682602 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:51.682924 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:51.683356 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:51.684376 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:51.685250 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:51.685739 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:51.685914 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:51.686831 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:51.686836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:51.686847 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:51.687219 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:51.687270 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:51.687293 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:51.687428 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:51.687602 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:32:51.687734 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260926 12:32:51.688807 29349 log.cc:824] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:51.690809 29349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: No bootstrap required, opened a new log
I20260926 12:32:51.691128 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:51.694365 29349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } }
I20260926 12:32:51.694595 29349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:51.694671 29349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f527181d1cf419fbac0fa67518980bc, State: Initialized, Role: FOLLOWER
I20260926 12:32:51.695124 29349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } }
I20260926 12:32:51.695271 29349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:51.695338 29349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:51.695431 29349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:51.696457 29349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } }
I20260926 12:32:51.696784 29349 leader_election.cc:302] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f527181d1cf419fbac0fa67518980bc; no voters:
I20260926 12:32:51.696983 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:51.693029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49863 (local address 127.7.99.62:41331)
0926 12:32:51.693147 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:51.693153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:51.693349 (+ 196us) server_negotiation.cc:407] Connection header received
0926 12:32:51.693461 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:51.693465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:51.693501 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:51.693597 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:51.693855 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:51.694358 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:51.695251 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:51.695589 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:51.695696 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:51.696519 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:51.696523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:51.696526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:51.696537 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:51.696568 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:51.696596 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:51.696679 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:51.696820 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 12:32:51.696853 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:32:51.697147 29349 leader_election.cc:288] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:51.697461 29360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:51.697719 29360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [term 1 LEADER]: Becoming Leader. State: Replica: 6f527181d1cf419fbac0fa67518980bc, State: Running, Role: LEADER
I20260926 12:32:51.698087 29360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } }
I20260926 12:32:51.698408 29349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:51.699935 29361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f527181d1cf419fbac0fa67518980bc. Latest consensus state: current_term: 1 leader_uuid: "6f527181d1cf419fbac0fa67518980bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } } }
I20260926 12:32:51.699954 29359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f527181d1cf419fbac0fa67518980bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f527181d1cf419fbac0fa67518980bc" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 41331 } } }
I20260926 12:32:51.700155 29359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:51.700445 29361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc [sys.catalog]: This master's current role is: LEADER
W20260926 12:32:51.703116 29373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:51.703192 29373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:51.703287 29374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:51.703989 29374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:51.709313 29374 catalog_manager.cc:1347] Generated new cluster ID: 26cc3e979e074e8ab4e2c2df5bd5eed1
I20260926 12:32:51.709374 29374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:51.720098 29374 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:51.721302 29374 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:51.728910 29374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f527181d1cf419fbac0fa67518980bc: Generated new TSK 0
I20260926 12:32:51.729628 29374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:51.741869 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29281
W20260926 12:32:51.747292 29266 connection.cc:535] client connection to 127.7.99.62:41331 recv error: Network error: failed to read from TLS socket (remote: 127.7.99.62:41331): Connection reset by peer (error 104)
2026-09-26T12:32:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:51 dist-test-slave-3jw2 krb5kdc[29380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:51 dist-test-slave-3jw2 krb5kdc[29380](info): set up 2 sockets
Sep 26 12:32:51 dist-test-slave-3jw2 krb5kdc[29380](info): commencing operation
krb5kdc: starting...
W20260926 12:32:52.776587 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:52 dist-test-slave-3jw2 krb5kdc[29380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:52.898932 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45199
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:53.008718 29393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:53.008960 29393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:53.009001 29393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:53.012923 29393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:53.012971 29393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:53.012989 29393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:53.013005 29393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:53.013026 29393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:53.018038 29393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:45199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:53.018934 29393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:53.019894 29393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:53.028137 29398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:53.028129 29401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:53.028188 29393 server_base.cc:1034] running on GCE node
W20260926 12:32:53.028129 29399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:53.028913 29393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:53.029966 29393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:53.031152 29393 hybrid_clock.cc:648] HybridClock initialized: now 1790425973031123 us; error 40 us; skew 500 ppm
Sep 26 12:32:53 dist-test-slave-3jw2 krb5kdc[29380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425973, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:53.033861 29393 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:53.034787 29393 webserver.cc:460] Webserver started at http://127.7.99.62:45307/ using document root <none> and password file <none>
I20260926 12:32:53.035336 29393 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:53.035420 29393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:53.035642 29393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:53.037659 29393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "1d63809476f94d8c98ada24fa03e4d10"
format_stamp: "Formatted at 2026-09-26 12:32:53 on dist-test-slave-3jw2"
server_key: "2eb0c99f4d1a82c21e06ca26c3249312"
server_key_iv: "325cfeb01ec7e765f681d6d77ad82675"
server_key_version: "encryptionkey@0"
I20260926 12:32:53.038203 29393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "1d63809476f94d8c98ada24fa03e4d10"
format_stamp: "Formatted at 2026-09-26 12:32:53 on dist-test-slave-3jw2"
server_key: "2eb0c99f4d1a82c21e06ca26c3249312"
server_key_iv: "325cfeb01ec7e765f681d6d77ad82675"
server_key_version: "encryptionkey@0"
I20260926 12:32:53.041889 29393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 12:32:53.044106 29408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:53.045300 29393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:53.045428 29393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "1d63809476f94d8c98ada24fa03e4d10"
format_stamp: "Formatted at 2026-09-26 12:32:53 on dist-test-slave-3jw2"
server_key: "2eb0c99f4d1a82c21e06ca26c3249312"
server_key_iv: "325cfeb01ec7e765f681d6d77ad82675"
server_key_version: "encryptionkey@0"
I20260926 12:32:53.045528 29393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:53.049983 29393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:53.052749 29393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:53.052963 29393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:53.059742 29393 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:45199
I20260926 12:32:53.059769 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:45199 every 8 connection(s)
I20260926 12:32:53.060654 29393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:53.063233 29461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:53.064715 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29393
I20260926 12:32:53.064846 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:53.065106 7564 external_mini_cluster.cc:1428] Setting key 049ae3b56730a8e8342ce00ce90eb938
I20260926 12:32:53.069710 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10: Bootstrap starting.
I20260926 12:32:53.070425 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:53.066406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41885 (local address 127.7.99.62:45199)
0926 12:32:53.066789 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:53.066808 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:53.066855 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:32:53.067693 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:53.067716 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:53.068069 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:53.068444 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:53.069061 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:53.069066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:53.069080 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:53.069090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:53.069146 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:53.069170 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:53.069297 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:53.069360 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 12:32:53.069497 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260926 12:32:53.072342 29461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:53.073208 29461 log.cc:824] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:53.074193 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:53.075383 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10: No bootstrap required, opened a new log
I20260926 12:32:53.077754 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:53.076129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57971 (local address 127.7.99.62:45199)
0926 12:32:53.076269 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:53.076273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:53.076444 (+ 171us) server_negotiation.cc:407] Connection header received
0926 12:32:53.076572 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:53.076574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:53.076599 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:53.076662 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:53.077335 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:53.077338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:53.077340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:53.077350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:53.077373 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:53.077383 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:53.077460 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:53.077577 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 12:32:53.077606 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:32:53.079488 29461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } }
I20260926 12:32:53.079720 29461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:53.079777 29461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d63809476f94d8c98ada24fa03e4d10, State: Initialized, Role: FOLLOWER
I20260926 12:32:53.080217 29461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } }
I20260926 12:32:53.080376 29461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:53.080441 29461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:53.080534 29461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:53.081386 29461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } }
I20260926 12:32:53.081750 29461 leader_election.cc:302] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d63809476f94d8c98ada24fa03e4d10; no voters:
I20260926 12:32:53.082032 29461 leader_election.cc:288] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:53.082161 29471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:53.082418 29471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [term 1 LEADER]: Becoming Leader. State: Replica: 1d63809476f94d8c98ada24fa03e4d10, State: Running, Role: LEADER
I20260926 12:32:53.082744 29471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } }
I20260926 12:32:53.083267 29461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:53.084410 29472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d63809476f94d8c98ada24fa03e4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } } }
I20260926 12:32:53.084560 29472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:53.084894 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d63809476f94d8c98ada24fa03e4d10. Latest consensus state: current_term: 1 leader_uuid: "1d63809476f94d8c98ada24fa03e4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d63809476f94d8c98ada24fa03e4d10" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 45199 } } }
I20260926 12:32:53.085012 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:53.085680 29480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:53.088654 29480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:53.094704 29480 catalog_manager.cc:1347] Generated new cluster ID: 148cd1fdc2354f70818102eead8f1780
I20260926 12:32:53.094769 29480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:53.120765 29480 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:53.121129 29480 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:53.137254 29480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d63809476f94d8c98ada24fa03e4d10: Generated new TSK 0
I20260926 12:32:53.138111 29480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:53.151975 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29393
2026-09-26T12:32:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:53 dist-test-slave-3jw2 krb5kdc[29492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:53 dist-test-slave-3jw2 krb5kdc[29492](info): set up 2 sockets
Sep 26 12:32:53 dist-test-slave-3jw2 krb5kdc[29492](info): commencing operation
krb5kdc: starting...
W20260926 12:32:54.180729 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:54.303313 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44787
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:44787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:54.413408 29505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:54.413697 29505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:54.413774 29505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:54.417737 29505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:54.417829 29505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:54.417869 29505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:54.417927 29505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:54.417970 29505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:54.423173 29505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:44787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:54.424233 29505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:54.425266 29505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:54.433807 29511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:54.433818 29510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:54.433830 29513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:54.434104 29505 server_base.cc:1034] running on GCE node
I20260926 12:32:54.434759 29505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:54.435881 29505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:54.437170 29505 hybrid_clock.cc:648] HybridClock initialized: now 1790425974437140 us; error 49 us; skew 500 ppm
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425974, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:54.439904 29505 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:54.440831 29505 webserver.cc:460] Webserver started at http://127.7.99.62:33143/ using document root <none> and password file <none>
I20260926 12:32:54.441370 29505 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:54.441455 29505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:54.441679 29505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:54.443656 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "a168c09a9adc4effab2d32dbaaba360c"
format_stamp: "Formatted at 2026-09-26 12:32:54 on dist-test-slave-3jw2"
server_key: "8cf6a76628d72ae5e0304e05a5910631"
server_key_iv: "0800787584c386bdeff758c789ebf342"
server_key_version: "encryptionkey@0"
I20260926 12:32:54.444247 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "a168c09a9adc4effab2d32dbaaba360c"
format_stamp: "Formatted at 2026-09-26 12:32:54 on dist-test-slave-3jw2"
server_key: "8cf6a76628d72ae5e0304e05a5910631"
server_key_iv: "0800787584c386bdeff758c789ebf342"
server_key_version: "encryptionkey@0"
I20260926 12:32:54.447806 29505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:32:54.450114 29520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:54.451325 29505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:54.451457 29505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "a168c09a9adc4effab2d32dbaaba360c"
format_stamp: "Formatted at 2026-09-26 12:32:54 on dist-test-slave-3jw2"
server_key: "8cf6a76628d72ae5e0304e05a5910631"
server_key_iv: "0800787584c386bdeff758c789ebf342"
server_key_version: "encryptionkey@0"
I20260926 12:32:54.451571 29505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:54.468716 29505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:54.471602 29505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:54.471791 29505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:54.478988 29505 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:44787
I20260926 12:32:54.479044 29572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:44787 every 8 connection(s)
I20260926 12:32:54.479969 29505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:54.482731 29573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:54.488041 29573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c: Bootstrap starting.
I20260926 12:32:54.489118 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29505
I20260926 12:32:54.489275 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:54.489573 7564 external_mini_cluster.cc:1428] Setting key a6dc8d4c02fd00cfca1a642f8fbb2c1b
I20260926 12:32:54.491128 29573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:54.491870 29573 log.cc:824] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:54.494129 29573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c: No bootstrap required, opened a new log
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:54.498732 29573 raft_consensus.cc:348] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } }
I20260926 12:32:54.498986 29573 raft_consensus.cc:374] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:54.499058 29573 raft_consensus.cc:729] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a168c09a9adc4effab2d32dbaaba360c, State: Initialized, Role: FOLLOWER
I20260926 12:32:54.499538 29573 consensus_queue.cc:260] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } }
I20260926 12:32:54.499725 29573 raft_consensus.cc:388] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:54.499789 29573 raft_consensus.cc:482] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:54.499866 29573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:54.501097 29573 raft_consensus.cc:504] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } }
I20260926 12:32:54.501471 29573 leader_election.cc:302] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a168c09a9adc4effab2d32dbaaba360c; no voters:
I20260926 12:32:54.501839 29573 leader_election.cc:288] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:54.501902 29578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:54.502115 29578 raft_consensus.cc:686] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [term 1 LEADER]: Becoming Leader. State: Replica: a168c09a9adc4effab2d32dbaaba360c, State: Running, Role: LEADER
I20260926 12:32:54.502461 29578 consensus_queue.cc:237] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } }
I20260926 12:32:54.503145 29573 sys_catalog.cc:564] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:54.504220 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a168c09a9adc4effab2d32dbaaba360c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } } }
I20260926 12:32:54.504347 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:54.504709 29580 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a168c09a9adc4effab2d32dbaaba360c. Latest consensus state: current_term: 1 leader_uuid: "a168c09a9adc4effab2d32dbaaba360c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a168c09a9adc4effab2d32dbaaba360c" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 44787 } } }
I20260926 12:32:54.504858 29580 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:54.505142 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:54.490738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40393 (local address 127.7.99.62:44787)
0926 12:32:54.491157 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:54.491192 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:54.491247 (+ 55us) server_negotiation.cc:407] Connection header received
0926 12:32:54.491915 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:54.491955 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:54.492430 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:54.492786 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:54.493657 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:54.494678 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:54.495851 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:54.496376 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:54.496492 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:54.499418 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:54.499424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:54.499440 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:54.499470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:54.501555 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:54.502098 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:54.502106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:54.502113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:54.502284 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:54.502606 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:54.502609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:54.502610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:54.503010 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:54.503233 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:54.503845 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 12:32:54.504155 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":152,"threads_started":1}
I20260926 12:32:54.505721 29587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:54.507963 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:54.508739 29587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:54.514415 29587 catalog_manager.cc:1347] Generated new cluster ID: 120d1ea1802d42b3876d3f01707be02c
I20260926 12:32:54.514475 29587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:54.518579 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:54.509956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42099 (local address 127.7.99.62:44787)
0926 12:32:54.510073 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:54.510077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:54.510791 (+ 714us) server_negotiation.cc:407] Connection header received
0926 12:32:54.510925 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:54.510928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:54.510973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:54.511048 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:54.511840 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:54.512499 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:54.513388 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:54.513635 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:54.513739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:54.515269 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:54.515273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:54.515276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:54.515300 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:54.516954 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:54.517398 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:54.517402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:54.517404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:54.517492 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:54.517759 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:54.517762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:54.517764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:54.517970 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:54.518094 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:54.518310 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 12:32:54.518425 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:32:54.537374 29587 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:54.538520 29587 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:54.545163 29587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a168c09a9adc4effab2d32dbaaba360c: Generated new TSK 0
I20260926 12:32:54.545840 29587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:54.549765 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29505
2026-09-26T12:32:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29604](info): set up 2 sockets
Sep 26 12:32:54 dist-test-slave-3jw2 krb5kdc[29604](info): commencing operation
krb5kdc: starting...
W20260926 12:32:55.584630 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:55 dist-test-slave-3jw2 krb5kdc[29604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:55.702903 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43595
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:55.809561 29617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:55.809863 29617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:55.809940 29617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:55.813815 29617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:55.813894 29617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:55.813931 29617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:55.813989 29617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:55.814029 29617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:55.819078 29617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:55.820052 29617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:55.821064 29617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:55.829393 29622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:55.829404 29625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:55.829442 29617 server_base.cc:1034] running on GCE node
W20260926 12:32:55.829406 29623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:55.830193 29617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:55.831225 29617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:55.832378 29617 hybrid_clock.cc:648] HybridClock initialized: now 1790425975832366 us; error 27 us; skew 500 ppm
Sep 26 12:32:55 dist-test-slave-3jw2 krb5kdc[29604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425975, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:55.839003 29617 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:55.839869 29617 webserver.cc:460] Webserver started at http://127.7.99.62:34499/ using document root <none> and password file <none>
I20260926 12:32:55.840435 29617 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:55.840497 29617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:55.840657 29617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:55.842621 29617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "e48746852b7a43f8aacccbf9a97af888"
format_stamp: "Formatted at 2026-09-26 12:32:55 on dist-test-slave-3jw2"
server_key: "08b12aa6376f005d605b6a443189e911"
server_key_iv: "bbd14d8ffb15089d8f33ccea580cc5e9"
server_key_version: "encryptionkey@0"
I20260926 12:32:55.843115 29617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "e48746852b7a43f8aacccbf9a97af888"
format_stamp: "Formatted at 2026-09-26 12:32:55 on dist-test-slave-3jw2"
server_key: "08b12aa6376f005d605b6a443189e911"
server_key_iv: "bbd14d8ffb15089d8f33ccea580cc5e9"
server_key_version: "encryptionkey@0"
I20260926 12:32:55.846711 29617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:32:55.849044 29632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:55.850257 29617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:32:55.850386 29617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "e48746852b7a43f8aacccbf9a97af888"
format_stamp: "Formatted at 2026-09-26 12:32:55 on dist-test-slave-3jw2"
server_key: "08b12aa6376f005d605b6a443189e911"
server_key_iv: "bbd14d8ffb15089d8f33ccea580cc5e9"
server_key_version: "encryptionkey@0"
I20260926 12:32:55.850494 29617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:55.875240 29617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:55.878587 29617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:55.878799 29617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:55.885864 29617 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43595
I20260926 12:32:55.885888 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43595 every 8 connection(s)
I20260926 12:32:55.886803 29617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:55.888481 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29617
I20260926 12:32:55.888607 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:55.888887 7564 external_mini_cluster.cc:1428] Setting key 229b008c1d452a774a71406e1ba3c33b
I20260926 12:32:55.889662 29685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:55.895145 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888: Bootstrap starting.
I20260926 12:32:55.897372 29685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:55.897414 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:55.890232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34921 (local address 127.7.99.62:43595)
0926 12:32:55.890653 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:55.890667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:55.890701 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:55.891324 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:55.891339 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:55.891651 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:55.892006 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:55.892429 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:55.893392 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:55.894293 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:55.894775 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:55.894937 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:55.895779 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:55.895784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:55.895796 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:55.896213 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:55.896283 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:55.896307 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:55.896408 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:55.896526 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 12:32:55.896740 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":144,"threads_started":1}
I20260926 12:32:55.898272 29685 log.cc:824] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:55.900281 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:55.900461 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888: No bootstrap required, opened a new log
I20260926 12:32:55.904522 29685 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } }
I20260926 12:32:55.904804 29685 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:55.904868 29685 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48746852b7a43f8aacccbf9a97af888, State: Initialized, Role: FOLLOWER
I20260926 12:32:55.905333 29685 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } }
I20260926 12:32:55.905517 29685 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:55.905587 29685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:55.905700 29685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:55.906447 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:55.902195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59179 (local address 127.7.99.62:43595)
0926 12:32:55.902314 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:55.902320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:55.902507 (+ 187us) server_negotiation.cc:407] Connection header received
0926 12:32:55.902601 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:55.902604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:55.902656 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:55.902741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:55.903061 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:55.903633 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:55.904737 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:55.905099 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:55.905226 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:55.906052 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:55.906054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:55.906056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:55.906067 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:55.906088 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:55.906098 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:55.906155 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:55.906264 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 12:32:55.906304 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:55.906876 29685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } }
I20260926 12:32:55.907229 29685 leader_election.cc:302] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48746852b7a43f8aacccbf9a97af888; no voters:
I20260926 12:32:55.907523 29685 leader_election.cc:288] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:55.907627 29695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:55.907835 29695 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [term 1 LEADER]: Becoming Leader. State: Replica: e48746852b7a43f8aacccbf9a97af888, State: Running, Role: LEADER
I20260926 12:32:55.908210 29695 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } }
I20260926 12:32:55.908725 29685 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:55.909735 29697 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48746852b7a43f8aacccbf9a97af888. Latest consensus state: current_term: 1 leader_uuid: "e48746852b7a43f8aacccbf9a97af888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } } }
I20260926 12:32:55.909865 29697 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:55.909989 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48746852b7a43f8aacccbf9a97af888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48746852b7a43f8aacccbf9a97af888" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43595 } } }
I20260926 12:32:55.910154 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:55.911080 29704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:55.914074 29704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:55.918882 29704 catalog_manager.cc:1347] Generated new cluster ID: 7f1e6783664c4e9fae614f10166b35f6
I20260926 12:32:55.918957 29704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:55.927352 29704 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:55.928273 29704 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:55.948699 29704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48746852b7a43f8aacccbf9a97af888: Generated new TSK 0
I20260926 12:32:55.949585 29704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:55.960815 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29617
2026-09-26T12:32:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:56 dist-test-slave-3jw2 krb5kdc[29716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:56 dist-test-slave-3jw2 krb5kdc[29716](info): set up 2 sockets
Sep 26 12:32:56 dist-test-slave-3jw2 krb5kdc[29716](info): commencing operation
krb5kdc: starting...
W20260926 12:32:57.004884 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:57 dist-test-slave-3jw2 krb5kdc[29716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:57.126744 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46155
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:57.237800 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:57.238118 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:57.238189 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:57.242172 29729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:57.242260 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:57.242300 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:57.242352 29729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:57.242391 29729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:57.247485 29729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:57.248528 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:57.249536 29729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:57.257825 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:57.257901 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:57.257901 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:57.258026 29729 server_base.cc:1034] running on GCE node
I20260926 12:32:57.258622 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:57.259651 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:57.260825 29729 hybrid_clock.cc:648] HybridClock initialized: now 1790425977260818 us; error 29 us; skew 500 ppm
Sep 26 12:32:57 dist-test-slave-3jw2 krb5kdc[29716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425977, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:57.263447 29729 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:57.264353 29729 webserver.cc:460] Webserver started at http://127.7.99.62:37561/ using document root <none> and password file <none>
I20260926 12:32:57.264879 29729 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:57.264961 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:57.265175 29729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:57.267104 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "d67f3052db1d48dba700c75fb07b0165"
format_stamp: "Formatted at 2026-09-26 12:32:57 on dist-test-slave-3jw2"
server_key: "1d6839346a87a21fc8154f8eea2e88d2"
server_key_iv: "a6197bd52a48fe6ef5113ad85cb51141"
server_key_version: "encryptionkey@0"
I20260926 12:32:57.267666 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "d67f3052db1d48dba700c75fb07b0165"
format_stamp: "Formatted at 2026-09-26 12:32:57 on dist-test-slave-3jw2"
server_key: "1d6839346a87a21fc8154f8eea2e88d2"
server_key_iv: "a6197bd52a48fe6ef5113ad85cb51141"
server_key_version: "encryptionkey@0"
I20260926 12:32:57.271183 29729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:32:57.273459 29744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:57.274624 29729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:57.274755 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "d67f3052db1d48dba700c75fb07b0165"
format_stamp: "Formatted at 2026-09-26 12:32:57 on dist-test-slave-3jw2"
server_key: "1d6839346a87a21fc8154f8eea2e88d2"
server_key_iv: "a6197bd52a48fe6ef5113ad85cb51141"
server_key_version: "encryptionkey@0"
I20260926 12:32:57.274866 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:57.279829 29729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:57.283051 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:57.283258 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:57.290036 29729 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:46155
I20260926 12:32:57.290062 29796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:46155 every 8 connection(s)
I20260926 12:32:57.290913 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:57.292456 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29729
I20260926 12:32:57.292613 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:57.292917 7564 external_mini_cluster.cc:1428] Setting key 3742131e40ad8835e23f65a4c004a2f8
I20260926 12:32:57.293995 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:57.297910 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:57.294305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53615 (local address 127.7.99.62:46155)
0926 12:32:57.294752 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:57.294768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:57.294802 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:57.295417 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:57.295432 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:57.295696 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:57.296056 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:57.296799 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:57.296804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:57.296822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:57.296832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:57.296881 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:57.296899 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:57.297040 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:57.297109 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 12:32:57.297254 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260926 12:32:57.299805 29797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165: Bootstrap starting.
I20260926 12:32:57.300508 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:57.302071 29797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:57.303004 29797 log.cc:824] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:57.304010 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:57.302375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33789 (local address 127.7.99.62:46155)
0926 12:32:57.302499 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:57.302504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:57.302654 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:32:57.302778 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:57.302780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:57.302806 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:57.302867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:57.303585 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:57.303589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:57.303592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:32:57.303604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:57.303630 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:32:57.303646 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:57.303736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:57.303841 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 12:32:57.303871 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:32:57.305148 29797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165: No bootstrap required, opened a new log
I20260926 12:32:57.308395 29797 raft_consensus.cc:348] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } }
I20260926 12:32:57.308594 29797 raft_consensus.cc:374] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:32:57.308635 29797 raft_consensus.cc:729] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67f3052db1d48dba700c75fb07b0165, State: Initialized, Role: FOLLOWER
I20260926 12:32:57.308961 29797 consensus_queue.cc:260] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } }
I20260926 12:32:57.309068 29797 raft_consensus.cc:388] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:32:57.309110 29797 raft_consensus.cc:482] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:32:57.309175 29797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:57.309926 29797 raft_consensus.cc:504] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } }
I20260926 12:32:57.310222 29797 leader_election.cc:302] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d67f3052db1d48dba700c75fb07b0165; no voters:
I20260926 12:32:57.310529 29797 leader_election.cc:288] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:32:57.310655 29807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:57.310887 29807 raft_consensus.cc:686] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [term 1 LEADER]: Becoming Leader. State: Replica: d67f3052db1d48dba700c75fb07b0165, State: Running, Role: LEADER
I20260926 12:32:57.311224 29807 consensus_queue.cc:237] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } }
I20260926 12:32:57.311744 29797 sys_catalog.cc:564] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:57.312861 29809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67f3052db1d48dba700c75fb07b0165. Latest consensus state: current_term: 1 leader_uuid: "d67f3052db1d48dba700c75fb07b0165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } } }
I20260926 12:32:57.312870 29808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d67f3052db1d48dba700c75fb07b0165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f3052db1d48dba700c75fb07b0165" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46155 } } }
I20260926 12:32:57.313007 29808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:57.313271 29809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:57.313408 29816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:57.316478 29816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:57.321970 29816 catalog_manager.cc:1347] Generated new cluster ID: bca363e73c9f400b9bfb0bab6fe80374
I20260926 12:32:57.322038 29816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:57.339262 29816 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:57.339689 29816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:57.348060 29816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d67f3052db1d48dba700c75fb07b0165: Generated new TSK 0
I20260926 12:32:57.348762 29816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:32:57.353353 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29729
2026-09-26T12:32:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1391 ms)
[----------] 18 tests from AuthTokenIssuingTest (25377 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:32:57 dist-test-slave-3jw2 krb5kdc[29828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:32:57 dist-test-slave-3jw2 krb5kdc[29828](info): set up 2 sockets
Sep 26 12:32:57 dist-test-slave-3jw2 krb5kdc[29828](info): commencing operation
krb5kdc: starting...
W20260926 12:32:58.412675 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:32:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:32:58.534839 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43569
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:58.643031 29841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:32:58.643282 29841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:58.643306 29841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:58.643378 29841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:58.647298 29841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:58.647354 29841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:58.647373 29841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:58.647389 29841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:58.647404 29841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:58.652653 29841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:43569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:58.653681 29841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:58.654682 29841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:58.662946 29847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:58.663060 29846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:58.663105 29849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:58.663228 29841 server_base.cc:1034] running on GCE node
I20260926 12:32:58.663743 29841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:58.664827 29841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:58.665982 29841 hybrid_clock.cc:648] HybridClock initialized: now 1790425978665965 us; error 28 us; skew 500 ppm
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:58.668675 29841 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:32:58.669493 29841 webserver.cc:460] Webserver started at http://127.7.99.62:42411/ using document root <none> and password file <none>
I20260926 12:32:58.670037 29841 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:58.670095 29841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:58.670324 29841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:58.672364 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "e7884e3e3ff045b8960128939521a1b9"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "faf322bc21988efc4d1433522f04a8d3"
server_key_iv: "67bacb719677ee7980cdd1f78f655cd0"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.672923 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "e7884e3e3ff045b8960128939521a1b9"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "faf322bc21988efc4d1433522f04a8d3"
server_key_iv: "67bacb719677ee7980cdd1f78f655cd0"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.676493 29841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:32:58.678777 29856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:58.679961 29841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:32:58.680090 29841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "e7884e3e3ff045b8960128939521a1b9"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "faf322bc21988efc4d1433522f04a8d3"
server_key_iv: "67bacb719677ee7980cdd1f78f655cd0"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.680238 29841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:58.705684 29841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:58.709438 29841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:58.709643 29841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:58.717775 29841 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:43569
I20260926 12:32:58.718130 29908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:43569 every 8 connection(s)
I20260926 12:32:58.718739 29841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:32:58.720849 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29841
I20260926 12:32:58.720973 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:32:58.721328 7564 external_mini_cluster.cc:1428] Setting key d0d908960bb2a4d6673e1978052e82f9
I20260926 12:32:58.721814 29909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:58.726801 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.61@KRBTEST.COM" created.
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:58.741736 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.730247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.7.99.62:43569)
0926 12:32:58.730508 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:58.730522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:58.730577 (+ 55us) server_negotiation.cc:407] Connection header received
0926 12:32:58.731110 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:58.731114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:58.731300 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:58.731441 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:58.732303 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.733515 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.734462 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.734852 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.735012 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.737359 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:58.737365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:58.737374 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:58.737393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:58.738920 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.739361 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.739371 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.739373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.739466 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.739766 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.739769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.739771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.740134 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:58.740299 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:58.740730 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 12:32:58.740956 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260926 12:32:58.741735 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.729257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43569 (local address 127.0.0.1:36090)
0926 12:32:58.729792 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.729834 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:58.730794 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:58.731465 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:58.731469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:58.731781 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:58.732112 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.732129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.733634 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.733637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.734350 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.734359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.734993 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.734997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.735192 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.735791 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:58.735806 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:58.737225 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:58.739023 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.739033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.739042 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.739259 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.739561 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.739564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.739566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.739671 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.740359 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:58.740368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:58.740503 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 12:32:58.740655 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 12:32:58.741028 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":212,"thread_start_us":149,"threads_started":1}
I20260926 12:32:58.744225 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:32:58.744999 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.744696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35318)
0926 12:32:58.744862 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.744946 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 12:32:58.745218 29857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
I20260926 12:32:58.746022 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.745836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35326)
0926 12:32:58.745935 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.745976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:32:58.746196 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:35341: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
I20260926 12:32:58.747748 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:35341
--webserver_interface=127.7.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:32:58.795436 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } attempt: 1
I20260926 12:32:58.796350 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.796017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35336)
0926 12:32:58.796207 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.796272 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 12:32:58.797114 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.796884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35340)
0926 12:32:58.796988 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.797046 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:32:58.797226 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:35341: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
W20260926 12:32:58.854489 29916 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:32:58.854797 29916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:58.854863 29916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:58.854939 29916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:58.858907 29916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:58.858996 29916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:58.859027 29916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:58.859057 29916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:58.859079 29916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:58.864487 29916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:58.865425 29916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:58.865393 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } attempt: 2
I20260926 12:32:58.866254 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.865930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35342)
0926 12:32:58.866096 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.866166 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:32:58.866564 29916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:32:58.867015 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.866786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35354)
0926 12:32:58.866896 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.866950 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:32:58.867148 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:35341: Network error: Client connection negotiation failed: client connection to 127.7.99.61:35341: connect: Connection refused (error 111)
W20260926 12:32:58.875041 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:58.875051 29924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:58.875203 29916 server_base.cc:1034] running on GCE node
W20260926 12:32:58.875109 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:58.875821 29916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:58.876962 29916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:58.878115 29916 hybrid_clock.cc:648] HybridClock initialized: now 1790425978878097 us; error 32 us; skew 500 ppm
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:58.880882 29916 init.cc:377] Logged in from keytab as kudu/127.7.99.61@KRBTEST.COM (short username kudu)
I20260926 12:32:58.881763 29916 webserver.cc:460] Webserver started at http://127.7.99.61:33329/ using document root <none> and password file <none>
I20260926 12:32:58.882308 29916 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:58.882367 29916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:58.882599 29916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:58.884740 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data/instance:
uuid: "75d01648f63c4d188dea64285e9021f3"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "3d51a10cba5d743d27b0ae0059770647"
server_key_iv: "ed39ec707af027a786583ac134e6ef87"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.885299 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal/instance:
uuid: "75d01648f63c4d188dea64285e9021f3"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "3d51a10cba5d743d27b0ae0059770647"
server_key_iv: "ed39ec707af027a786583ac134e6ef87"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.889051 29916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:32:58.891352 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:58.892549 29916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:32:58.892684 29916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "75d01648f63c4d188dea64285e9021f3"
format_stamp: "Formatted at 2026-09-26 12:32:58 on dist-test-slave-3jw2"
server_key: "3d51a10cba5d743d27b0ae0059770647"
server_key_iv: "ed39ec707af027a786583ac134e6ef87"
server_key_version: "encryptionkey@0"
I20260926 12:32:58.892788 29916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:58.913308 29916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:58.916404 29916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:58.916589 29916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:58.923446 29916 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.61:35341
I20260926 12:32:58.923744 29983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.61:35341 every 8 connection(s)
I20260926 12:32:58.924441 29916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
I20260926 12:32:58.927112 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:58.930892 29984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:32:58.933686 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29916
I20260926 12:32:58.933813 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-1/wal/instance
I20260926 12:32:58.934084 7564 external_mini_cluster.cc:1428] Setting key 177b8b2690775e170d9a842a735d2c6d
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.99.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.60@KRBTEST.COM" created.
I20260926 12:32:58.944665 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.932665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.7.99.62:43569)
0926 12:32:58.932819 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:58.932824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:58.933653 (+ 829us) server_negotiation.cc:407] Connection header received
0926 12:32:58.934649 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:58.934652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:58.934724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:58.934820 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:58.935911 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.936496 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.937589 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.937820 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.937929 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.940765 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:58.940768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:58.940770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:58.940793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:58.942208 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.942893 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.942896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.942897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.943011 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.943464 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.943466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.943468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.943686 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:58.943770 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:58.944280 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 12:32:58.944423 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:32:58.945592 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.933081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43569 (local address 127.0.0.1:36094)
0926 12:32:58.933513 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.933555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:58.934452 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:58.934982 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:58.934989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:58.935313 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:58.935775 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.935786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.936615 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.936623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.937466 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.937474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.937974 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.937978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.938136 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.938779 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:58.938794 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:58.940592 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:58.942346 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.942351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.942365 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.942776 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.943137 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.943142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.943145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.943352 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.943778 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:58.943783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:58.944011 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 12:32:58.944220 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 12:32:58.944562 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260926 12:32:58.947024 29984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
I20260926 12:32:58.955857 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.947517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35356)
0926 12:32:58.947737 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.947753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:58.947827 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:58.948292 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:58.948294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:58.948457 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:58.948529 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.948534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.949467 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.949470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.949992 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.949999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.950556 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.950559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.950649 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.951129 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:58.951139 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:58.952414 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:58.954397 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.954402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.954405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.954601 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.954884 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.954886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.954887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.955005 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.955375 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:58.955377 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:58.955452 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 12:32:58.955588 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 12:32:58.955745 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 12:32:58.956001 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.947605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.7.99.61:35341)
0926 12:32:58.947923 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:58.947933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:58.947951 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:32:58.947984 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:58.947988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:58.948134 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:58.948318 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:58.948645 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.949358 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.950108 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.950451 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.950561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.952553 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:58.952561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:58.952571 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:58.952596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:58.954291 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.954714 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.954719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.954720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.954799 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.955071 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.955073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.955074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.955294 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:58.955392 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:58.955638 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:32:58.955819 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":221,"thread_start_us":153,"threads_started":1}
I20260926 12:32:58.957903 29984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:32:58.958380 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } attempt: 3
I20260926 12:32:58.958758 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.958509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58472)
0926 12:32:58.958600 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.958675 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 12:32:58.959005 29932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
I20260926 12:32:58.959978 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.959789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58484)
0926 12:32:58.959886 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.959933 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 12:32:58.960099 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:36395
--webserver_interface=127.7.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:32:58.960116 29984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Sep 26 12:32:58 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
I20260926 12:32:58.967612 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.958832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.7.99.61:35341)
0926 12:32:58.958930 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:58.958934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:58.959008 (+ 74us) server_negotiation.cc:407] Connection header received
0926 12:32:58.959152 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:58.959157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:58.959197 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:58.959286 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:58.959744 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.960651 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.961900 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.962287 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.962405 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.964594 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:58.964597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:58.964599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:58.964618 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:58.965992 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.966501 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.966503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.966505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.966573 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:58.966872 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:58.966874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:58.966875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:58.967023 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:58.967092 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:58.967430 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 12:32:58.967515 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:32:58.967867 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.958775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35368)
0926 12:32:58.958953 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.958966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:58.959052 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:58.959303 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:58.959306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:58.959494 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:58.959608 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.959617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.960794 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.960797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.961785 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:58.961795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:58.962391 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:58.962393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:58.962467 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:58.962933 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:58.962943 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:58.964466 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:58.966105 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.966111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.966115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.966404 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.966666 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:58.966672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:58.966676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:58.966784 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:58.967114 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:58.967118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:58.967246 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 12:32:58.967377 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 12:32:58.967539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 12:32:58.968999 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:32:58.969760 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.969461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58498)
0926 12:32:58.969595 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.969661 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:32:58.970500 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:58.970199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58502)
0926 12:32:58.970362 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:58.970423 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 12:32:58.970633 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
I20260926 12:32:59.001811 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 1
I20260926 12:32:59.002671 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.002344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58512)
0926 12:32:59.002507 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.002577 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 12:32:59.003456 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.003181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58520)
0926 12:32:59.003303 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.003370 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:32:59.003610 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
I20260926 12:32:59.009366 29984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 1
I20260926 12:32:59.010006 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.009764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58524)
0926 12:32:59.009887 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.009943 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 12:32:59.010627 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.010428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58534)
0926 12:32:59.010528 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.010572 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:32:59.010756 29984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
W20260926 12:32:59.071751 29991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:32:59.072131 29991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:32:59.072220 29991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:32:59.072291 29991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:32:59.076223 29991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:32:59.076292 29991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:32:59.076327 29991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:32:59.076361 29991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:32:59.076395 29991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:32:59.078785 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 2
I20260926 12:32:59.079007 29984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 2
I20260926 12:32:59.079598 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.079275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58542)
0926 12:32:59.079442 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.079505 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:32:59.079780 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.079502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58558)
0926 12:32:59.079646 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.079705 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 12:32:59.080499 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.080275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58568)
0926 12:32:59.080385 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.080437 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:32:59.080631 29984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
I20260926 12:32:59.080559 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.080324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58574)
0926 12:32:59.080433 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.080481 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:32:59.080751 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:36395: Network error: Client connection negotiation failed: client connection to 127.7.99.60:36395: connect: Connection refused (error 111)
I20260926 12:32:59.082232 29991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:43569,127.7.99.61:35341,127.7.99.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:32:59.083134 29991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:32:59.084092 29991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:32:59.092047 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:59.091957 29997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:32:59.091950 29996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:32:59.092455 29991 server_base.cc:1034] running on GCE node
I20260926 12:32:59.092985 29991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:32:59.093969 29991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:32:59.095139 29991 hybrid_clock.cc:648] HybridClock initialized: now 1790425979095089 us; error 57 us; skew 500 ppm
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425979, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:32:59.098579 29991 init.cc:377] Logged in from keytab as kudu/127.7.99.60@KRBTEST.COM (short username kudu)
I20260926 12:32:59.099541 29991 webserver.cc:460] Webserver started at http://127.7.99.60:32897/ using document root <none> and password file <none>
I20260926 12:32:59.100098 29991 fs_manager.cc:359] Metadata directory not provided
I20260926 12:32:59.100208 29991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:32:59.100471 29991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:32:59.102454 29991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data/instance:
uuid: "af67ad7dac0f4babb5339ce19c17ddda"
format_stamp: "Formatted at 2026-09-26 12:32:59 on dist-test-slave-3jw2"
server_key: "3032b69d3a02b7a8372311aebba6b662"
server_key_iv: "1bb907446ea177198c9caaa0d49ab623"
server_key_version: "encryptionkey@0"
I20260926 12:32:59.103003 29991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal/instance:
uuid: "af67ad7dac0f4babb5339ce19c17ddda"
format_stamp: "Formatted at 2026-09-26 12:32:59 on dist-test-slave-3jw2"
server_key: "3032b69d3a02b7a8372311aebba6b662"
server_key_iv: "1bb907446ea177198c9caaa0d49ab623"
server_key_version: "encryptionkey@0"
I20260926 12:32:59.106642 29991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:32:59.109320 30006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:32:59.110484 29991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:32:59.110636 29991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "af67ad7dac0f4babb5339ce19c17ddda"
format_stamp: "Formatted at 2026-09-26 12:32:59 on dist-test-slave-3jw2"
server_key: "3032b69d3a02b7a8372311aebba6b662"
server_key_iv: "1bb907446ea177198c9caaa0d49ab623"
server_key_version: "encryptionkey@0"
I20260926 12:32:59.110741 29991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:32:59.121912 29991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:32:59.124599 29991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:32:59.124802 29991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:32:59.131546 29991 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.60:36395
I20260926 12:32:59.131812 30058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.60:36395 every 8 connection(s)
I20260926 12:32:59.132566 29991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
I20260926 12:32:59.135385 30059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:32:59.136284 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 29991
I20260926 12:32:59.136425 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425700812969-7564-0/minicluster-data/master-2/wal/instance
I20260926 12:32:59.136757 7564 external_mini_cluster.cc:1428] Setting key 1a189cb710289d821d093b84918c9c48
I20260926 12:32:59.139559 30059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425979, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:59.150981 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.141647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.7.99.62:43569)
0926 12:32:59.141777 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.141782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.142053 (+ 271us) server_negotiation.cc:407] Connection header received
0926 12:32:59.142196 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.142200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.142251 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.142335 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.143107 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.143681 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.144720 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.145091 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.145220 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.147736 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.147739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.147741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.147763 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.149177 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.149655 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.149661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.149662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.149750 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.150019 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.150024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.150025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.150231 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.150334 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.150717 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.150825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:32:59.151224 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.141492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:43569 (local address 127.0.0.1:36110)
0926 12:32:59.141916 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.141971 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:59.142096 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:59.142399 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:59.142406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:59.142841 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:59.142982 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.142989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.143827 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.143832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.144602 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.144612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.145363 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.145366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.145458 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.145915 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:59.145939 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:59.147608 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:59.149313 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.149321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.149336 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.149557 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.149849 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.149852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.149854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.149935 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.150390 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:59.150396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:59.150500 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 12:32:59.150637 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 12:32:59.150819 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":166,"threads_started":1}
I20260926 12:32:59.151222 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.137994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.7.99.60:36395)
0926 12:32:59.138394 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.138408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.138442 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:32:59.139105 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.139121 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.139434 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.139706 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.140646 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.141814 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.142761 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.143363 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.143474 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.146355 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.146360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.146369 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.146391 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.148451 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.148882 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.148892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.148897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.148995 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.149275 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.149280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.149281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.149651 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.149860 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.150135 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.150401 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260926 12:32:59.152431 30059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425979, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:32:59.160364 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.152915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:35341 (local address 127.0.0.1:35372)
0926 12:32:59.153077 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.153093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:59.153169 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:59.153526 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:59.153530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:59.153705 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:59.153808 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.153820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.154519 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.154524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.155019 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.155026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.155483 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.155486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.155581 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.156006 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:59.156023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:59.157219 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:59.158914 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.158919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.158924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.159120 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.159421 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.159429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.159431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.159499 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.159867 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:59.159870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:59.159959 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 12:32:59.160061 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 12:32:59.160195 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 12:32:59.160396 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.153005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.7.99.61:35341)
0926 12:32:59.153153 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.153157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.153170 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:32:59.153257 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.153259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.153311 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.153391 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.153924 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.154414 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.155138 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.155365 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.155481 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.157360 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.157364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.157366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.157392 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.158797 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.159234 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.159239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.159241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.159326 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.159588 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.159593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.159594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.159776 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.159878 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.160115 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.160263 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:32:59.161068 30059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:32:59.161376 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.153331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.7.99.62:43569)
0926 12:32:59.153437 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.153440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.153452 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:32:59.153563 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.153569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.153615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.153703 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.154223 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.154683 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.155417 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.155637 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.155735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.158012 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.158015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.158017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.158037 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.159494 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.159895 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.159902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.159905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.159983 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.160281 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.160285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.160286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.160514 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.160748 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.161058 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.161220 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425979, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
I20260926 12:32:59.169840 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.161663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58592)
0926 12:32:59.161743 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.161756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:59.161823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:59.162113 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:59.162115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:59.162258 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:59.162366 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.162373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.163409 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.163413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.164269 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.164281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.164806 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.164808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.164866 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.165278 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:59.165289 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:59.166416 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:59.168231 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.168234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.168238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.168466 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.168727 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.168732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.168734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.168824 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.169228 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:59.169231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:59.169320 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 12:32:59.169438 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:32:59.169584 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 12:32:59.169858 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.161808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.7.99.60:36395)
0926 12:32:59.161931 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.161937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.161953 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:32:59.161987 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.161990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.162030 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.162103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.162477 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.163329 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.164368 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.164706 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.164808 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.166550 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.166554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.166557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.166590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.168094 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.168563 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.168565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.168566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.168637 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.168918 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.168923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.168926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.169126 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.169203 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.169516 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.169640 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":45}
I20260926 12:32:59.171291 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.162520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.7.99.61:35341)
0926 12:32:59.162634 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.162638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.162653 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:32:59.162691 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.162694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.162737 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.162820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.163363 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.164110 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.165149 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.165415 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.165497 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.168044 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.168048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.168050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.168073 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.169687 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.170175 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.170178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.170179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.170267 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.170525 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.170531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.170534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.170710 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.170910 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.171046 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.171148 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:32:59.171770 29984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 3
I20260926 12:32:59.173625 30059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: Bootstrap starting.
I20260926 12:32:59.174718 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:32:59.176487 30059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: Neither blocks nor log segments found. Creating new log.
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:32:59.175959 29883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e7884e3e3ff045b8960128939521a1b9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36116
I20260926 12:32:59.177366 30059 log.cc:824] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:59.179536 30059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: No bootstrap required, opened a new log
I20260926 12:32:59.179876 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } attempt: 3
I20260926 12:32:59.180095 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.172142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58602)
0926 12:32:59.172335 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.172349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:59.172420 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:59.172798 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:59.172801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:59.172961 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:59.173046 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.173051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.173960 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.173966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.174560 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.174568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.175131 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.175135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.175229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.175686 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:59.175701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:59.176879 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:59.178660 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.178665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.178667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.178856 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.179134 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.179138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.179140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.179223 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.179596 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:59.179602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:59.179699 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 12:32:59.179817 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:32:59.179945 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 12:32:59.181083 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.172236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.7.99.60:36395)
0926 12:32:59.172596 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.172601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.172613 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:32:59.172642 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.172645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.172679 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.172755 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.173171 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.173818 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.174666 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.174995 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.175415 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.177013 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.177016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.177018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.177044 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.178542 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.178959 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.178963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.178965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.179045 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.179310 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.179314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.179316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.179497 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.179672 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.180465 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.180614 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260926 12:32:59.183677 30059 raft_consensus.cc:348] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.183946 30059 raft_consensus.cc:729] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af67ad7dac0f4babb5339ce19c17ddda, State: Initialized, Role: FOLLOWER
I20260926 12:32:59.184005 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: Bootstrap starting.
I20260926 12:32:59.184306 30059 consensus_queue.cc:260] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.184787 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.184919 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 12:32:59 dist-test-slave-3jw2 krb5kdc[29828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425978, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:32:59.185643 30059 sys_catalog.cc:564] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:59.186780 29984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:59.187683 29984 log.cc:824] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:59.190163 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.180605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:36395 (local address 127.0.0.1:58618)
0926 12:32:59.180718 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 12:32:59.180731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:32:59.180837 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:32:59.181082 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:32:59.181085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:32:59.181274 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:32:59.181391 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.181398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.182489 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.182493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.183321 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:32:59.183332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.183909 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.183917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:32:59.184013 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.184514 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:32:59.184533 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:32:59.185962 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:32:59.188480 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.188483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.188485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.188690 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.189016 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:32:59.189020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:32:59.189025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:32:59.189130 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:32:59.189617 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:32:59.189619 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:32:59.189746 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 12:32:59.189863 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 12:32:59.190002 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:32:59.190361 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.180356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.7.99.60:36395)
0926 12:32:59.180686 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.180695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.180795 (+ 100us) server_negotiation.cc:407] Connection header received
0926 12:32:59.180938 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.180940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.180980 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.181057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.181632 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.182377 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.183448 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.183776 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.183873 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.186129 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.186133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.186135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.186159 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.188155 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.188821 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.188826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.188828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.188923 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.189256 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.189259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.189260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.189471 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.189561 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.189929 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.190048 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
W20260926 12:32:59.191107 30077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:59.191174 30077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:59.192447 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: No bootstrap required, opened a new log
I20260926 12:32:59.193086 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: Bootstrap starting.
I20260926 12:32:59.190685 29991 master_runner.cc:186] Error getting master registration for 127.7.99.62:43569: OK, instance_id { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" instance_seqno: 1790425978715837 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:32:59.193614 29991 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:32:59.195566 29909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: Neither blocks nor log segments found. Creating new log.
I20260926 12:32:59.195765 29984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.195986 29984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d01648f63c4d188dea64285e9021f3, State: Initialized, Role: FOLLOWER
I20260926 12:32:59.196337 29984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.196403 29909 log.cc:824] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:32:59.196864 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.197005 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.197547 29984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:59.198410 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: No bootstrap required, opened a new log
I20260926 12:32:59.201342 29909 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.201537 29909 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7884e3e3ff045b8960128939521a1b9, State: Initialized, Role: FOLLOWER
I20260926 12:32:59.201828 29909 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
W20260926 12:32:59.202692 30092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:59.202759 30092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:59.202986 29909 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:32:59.200898 29916 master_runner.cc:186] Error getting master registration for 127.7.99.62:43569: OK, instance_id { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" instance_seqno: 1790425978715837 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:32:59.204231 29916 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:32:59.204456 29883 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:32:59.204576 29883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:59.205911 29883 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.207135 30094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.207286 30094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.206857 29883 leader_election.cc:288] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [CANDIDATE]: Term 1 election: Requested vote from peers 75d01648f63c4d188dea64285e9021f3 (127.7.99.61:35341), af67ad7dac0f4babb5339ce19c17ddda (127.7.99.60:36395)
I20260926 12:32:59.208410 30033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7884e3e3ff045b8960128939521a1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af67ad7dac0f4babb5339ce19c17ddda"
I20260926 12:32:59.208638 30033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:32:59.210058 30033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7884e3e3ff045b8960128939521a1b9 in term 1.
I20260926 12:32:59.210984 29958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7884e3e3ff045b8960128939521a1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "75d01648f63c4d188dea64285e9021f3"
I20260926 12:32:59.211223 29958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:32:59.211846 29862 tablet.cc:2369] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:32:59.212546 29958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7884e3e3ff045b8960128939521a1b9 in term 1.
I20260926 12:32:59.212724 29857 leader_election.cc:302] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af67ad7dac0f4babb5339ce19c17ddda, e7884e3e3ff045b8960128939521a1b9; no voters:
I20260926 12:32:59.212985 30094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:32:59.213219 30094 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [term 1 LEADER]: Becoming Leader. State: Replica: e7884e3e3ff045b8960128939521a1b9, State: Running, Role: LEADER
I20260926 12:32:59.213537 30094 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } }
I20260926 12:32:59.215428 30110 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7884e3e3ff045b8960128939521a1b9. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.215605 30110 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: This master's current role is: LEADER
W20260926 12:32:59.218319 30105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:32:59.218398 30105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:32:59.218403 29841 master_runner.cc:263] No leader master found from master e7884e3e3ff045b8960128939521a1b9
I20260926 12:32:59.218477 29841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:32:59.224503 30115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:32:59.231411 30115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:32:59.238900 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.214564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.7.99.61:35341)
0926 12:32:59.214674 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.214678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.224829 (+ 10151us) server_negotiation.cc:407] Connection header received
0926 12:32:59.224876 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.224878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.224924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.225002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.226042 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.226841 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.228057 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.228425 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.228505 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.232977 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.232981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.232984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.233013 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.234860 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.236394 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.236401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.236408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.236499 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.237867 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.237873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.237876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.238103 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.238234 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.238573 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.238702 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:32:59.239154 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.215280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.7.99.60:36395)
0926 12:32:59.215380 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.215384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.224999 (+ 9615us) server_negotiation.cc:407] Connection header received
0926 12:32:59.225103 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.225106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.225149 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.225225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.226042 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.226848 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.233335 (+ 6487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.233772 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.233877 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.235585 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.235588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.235591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.235620 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.237331 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.237825 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.237831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.237836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.237922 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.238210 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.238213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.238215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.238417 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.238531 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.238874 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.239006 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 12:32:59.247675 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:32:59.214383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.7.99.62:43569)
0926 12:32:59.214502 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.214507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.224440 (+ 9933us) server_negotiation.cc:407] Connection header received
0926 12:32:59.224583 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.224586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.224636 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.224726 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.235421 (+ 10695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.236254 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.242082 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:32:59.242437 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:32:59.242509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:32:59.244237 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:32:59.244241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:32:59.244244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:32:59.244271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:32:59.245914 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.246394 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.246397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.246399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.246478 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:32:59.246756 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:32:59.246759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:32:59.246761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:32:59.246962 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:32:59.247062 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:32:59.247403 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 12:32:59.247522 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:32:59.253011 29958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [term 1 FOLLOWER]: Refusing update from remote peer e7884e3e3ff045b8960128939521a1b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:59.253011 30033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [term 1 FOLLOWER]: Refusing update from remote peer e7884e3e3ff045b8960128939521a1b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:32:59.253731 30110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:59.255923 30094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:32:59.260233 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7884e3e3ff045b8960128939521a1b9. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.260354 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.261125 30110 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.261140 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7884e3e3ff045b8960128939521a1b9. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.261251 30110 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:59.261268 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.262526 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.262637 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.262988 30117 mvcc.cc:205] Tried to move back new op lower bound from 7333584811011616768 to 7333584810862125056. Current Snapshot: MvccSnapshot[applied={T|T < 7333584811011616768}]
I20260926 12:32:59.264962 30118 mvcc.cc:205] Tried to move back new op lower bound from 7333584811011616768 to 7333584810862125056. Current Snapshot: MvccSnapshot[applied={T|T < 7333584811011616768}]
I20260926 12:32:59.268435 30094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.268558 30094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:32:59.269543 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7884e3e3ff045b8960128939521a1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7884e3e3ff045b8960128939521a1b9" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 43569 } } peers { permanent_uuid: "75d01648f63c4d188dea64285e9021f3" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 35341 } } peers { permanent_uuid: "af67ad7dac0f4babb5339ce19c17ddda" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 36395 } } }
I20260926 12:32:59.269675 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:32:59.269963 30120 mvcc.cc:205] Tried to move back new op lower bound from 7333584811011616768 to 7333584810862125056. Current Snapshot: MvccSnapshot[applied={T|T < 7333584811011616768}]
I20260926 12:32:59.270485 30115 catalog_manager.cc:1347] Generated new cluster ID: ed1c9d5b24cf433c8a14eb8563261f10
I20260926 12:32:59.270547 30115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:32:59.307816 30115 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:32:59.310757 30115 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:32:59.331547 30115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7884e3e3ff045b8960128939521a1b9: Generated new TSK 0
I20260926 12:32:59.332509 30115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:32:59.371194 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.368153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.7.99.62:43569)
0926 12:32:59.368309 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.368313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.370090 (+ 1777us) server_negotiation.cc:407] Connection header received
0926 12:32:59.370286 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.370290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.370344 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.370397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.371060 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36128: BlockingRecv error: recv got EOF from 127.0.0.1:36128 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 12:32:59.372617 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.368905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.7.99.61:35341)
0926 12:32:59.369017 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.369022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.371668 (+ 2646us) server_negotiation.cc:407] Connection header received
0926 12:32:59.371856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.371859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.371908 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.371960 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.372505 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35408: BlockingRecv error: recv got EOF from 127.0.0.1:35408 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 12:32:59.374120 30065 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.369503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.7.99.60:36395)
0926 12:32:59.369605 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.369609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.373002 (+ 3393us) server_negotiation.cc:407] Connection header received
0926 12:32:59.373303 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.373307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.373359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.373409 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.374008 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58630: BlockingRecv error: recv got EOF from 127.0.0.1:58630 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 12:32:59.378332 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.376935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.7.99.61:35341)
0926 12:32:59.377036 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.377040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.377438 (+ 398us) server_negotiation.cc:407] Connection header received
0926 12:32:59.377635 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.377639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.377721 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.377772 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.378243 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35414: BlockingRecv error: recv got EOF from 127.0.0.1:35414 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 12:32:59.383266 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.381955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.7.99.61:35341)
0926 12:32:59.382050 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.382054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.382490 (+ 436us) server_negotiation.cc:407] Connection header received
0926 12:32:59.382648 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.382651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.382694 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.382754 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.383180 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35428: BlockingRecv error: recv got EOF from 127.0.0.1:35428 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 12:32:59.389128 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.387607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.7.99.61:35341)
0926 12:32:59.387715 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.387719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.388192 (+ 473us) server_negotiation.cc:407] Connection header received
0926 12:32:59.388372 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.388376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.388436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.388485 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.389034 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35440: BlockingRecv error: recv got EOF from 127.0.0.1:35440 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 12:32:59.396965 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.395226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.7.99.61:35341)
0926 12:32:59.395357 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.395361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.395937 (+ 576us) server_negotiation.cc:407] Connection header received
0926 12:32:59.396133 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.396137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.396207 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.396261 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.396852 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35444: BlockingRecv error: recv got EOF from 127.0.0.1:35444 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 12:32:59.413686 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.408725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.7.99.61:35341)
0926 12:32:59.408846 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.408851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.412531 (+ 3680us) server_negotiation.cc:407] Connection header received
0926 12:32:59.412780 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.412784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.412843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.412900 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.413555 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35450: BlockingRecv error: recv got EOF from 127.0.0.1:35450 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 12:32:59.434445 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.432527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.7.99.61:35341)
0926 12:32:59.432695 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.432702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.433364 (+ 662us) server_negotiation.cc:407] Connection header received
0926 12:32:59.433603 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.433607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.433666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.433723 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.434329 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35466: BlockingRecv error: recv got EOF from 127.0.0.1:35466 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 12:32:59.470148 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.468934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.7.99.61:35341)
0926 12:32:59.469074 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.469079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.469247 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:32:59.469349 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.469351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.469406 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.469477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.470015 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35474: BlockingRecv error: recv got EOF from 127.0.0.1:35474 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:32:59.537441 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.536225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.7.99.61:35341)
0926 12:32:59.536375 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.536380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.536488 (+ 108us) server_negotiation.cc:407] Connection header received
0926 12:32:59.536678 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.536681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.536736 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.536823 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.537316 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35480: BlockingRecv error: recv got EOF from 127.0.0.1:35480 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 12:32:59.669045 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.667723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.7.99.61:35341)
0926 12:32:59.667916 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.667921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.668101 (+ 180us) server_negotiation.cc:407] Connection header received
0926 12:32:59.668309 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.668313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.668367 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.668453 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.668921 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35494: BlockingRecv error: recv got EOF from 127.0.0.1:35494 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260926 12:32:59.928529 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:32:59.927409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.7.99.61:35341)
0926 12:32:59.927552 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:32:59.927557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:32:59.927674 (+ 117us) server_negotiation.cc:407] Connection header received
0926 12:32:59.927781 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:32:59.927784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:32:59.927835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:32:59.927930 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:32:59.928422 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35496: BlockingRecv error: recv got EOF from 127.0.0.1:35496 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:33:00.192564 30077 catalog_manager.cc:1259] Loaded cluster ID: ed1c9d5b24cf433c8a14eb8563261f10
I20260926 12:33:00.192662 30077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: loading cluster ID for follower catalog manager: success
I20260926 12:33:00.194314 30077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: acquiring CA information for follower catalog manager: success
I20260926 12:33:00.195474 30077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af67ad7dac0f4babb5339ce19c17ddda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:33:00.203948 30092 catalog_manager.cc:1259] Loaded cluster ID: ed1c9d5b24cf433c8a14eb8563261f10
I20260926 12:33:00.204054 30092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: loading cluster ID for follower catalog manager: success
I20260926 12:33:00.205783 30092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: acquiring CA information for follower catalog manager: success
I20260926 12:33:00.206939 30092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75d01648f63c4d188dea64285e9021f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:33:00.446678 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:00.442707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.7.99.61:35341)
0926 12:33:00.442965 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:00.442973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:00.443012 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:33:00.443196 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:00.443210 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:00.443338 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:00.443461 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:00.443720 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.444278 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.445140 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.445377 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.445486 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:00.445818 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:00.446267 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:00.446401 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:00.446461 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 12:33:00.446520 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20260926 12:33:00.453354 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:00.449667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.7.99.60:36395)
0926 12:33:00.449919 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:00.449924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:00.449939 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:00.450105 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:00.450108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:00.450236 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:00.450355 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:00.450555 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.451072 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.451908 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.452138 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.452271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:00.452550 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:00.452958 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:00.453088 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:00.453136 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 12:33:00.453189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260926 12:33:00.460440 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:00.456436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.7.99.61:35341)
0926 12:33:00.456541 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:00.456545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:00.456765 (+ 220us) server_negotiation.cc:407] Connection header received
0926 12:33:00.456934 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:00.456938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:00.456989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:00.457088 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:00.457344 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.458129 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.459252 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.459531 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.459628 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:00.459922 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:00.460011 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:00.460057 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:00.460240 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 12:33:00.460308 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:33:00.460489 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:00.456783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.7.99.60:36395)
0926 12:33:00.456878 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:00.456883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:00.457084 (+ 201us) server_negotiation.cc:407] Connection header received
0926 12:33:00.457237 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:00.457240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:00.457283 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:00.457374 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:00.457584 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.458296 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.459302 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.459625 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.459692 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:00.460023 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:00.460094 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:00.460200 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:00.460338 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 12:33:00.460383 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 12:33:00.460593 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:00.456094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.7.99.62:43569)
0926 12:33:00.456383 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:00.456387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:00.456403 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:33:00.456497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:00.456500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:00.456673 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:00.456793 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:00.457147 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.457852 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.458992 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:00.459378 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:00.459492 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:00.459841 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:00.460157 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:00.460270 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:00.460352 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 12:33:00.460407 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":138,"threads_started":1}
I20260926 12:33:00.463531 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29841
I20260926 12:33:00.469895 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29916
I20260926 12:33:00.475677 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 29991
2026-09-26T12:33:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3124 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:00 dist-test-slave-3jw2 krb5kdc[30207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:00 dist-test-slave-3jw2 krb5kdc[30207](info): set up 2 sockets
Sep 26 12:33:00 dist-test-slave-3jw2 krb5kdc[30207](info): commencing operation
krb5kdc: starting...
W20260926 12:33:01.529026 7564 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/test-user.keytab.
Sep 26 12:33:01 dist-test-slave-3jw2 krb5kdc[30207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
Entry for principal kudu/127.7.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab.
I20260926 12:33:01.647991 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46131
--webserver_interface=127.7.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:01.757331 30220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:33:01.757615 30220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:01.757639 30220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:01.757748 30220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:01.761839 30220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:01.761898 30220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:01.761917 30220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:01.761934 30220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:01.761996 30220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:01.767279 30220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.62.keytab
--principal=kudu/127.7.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.62:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:01.768390 30220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:01.769428 30220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:01.777884 30226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:01.777956 30220 server_base.cc:1034] running on GCE node
W20260926 12:33:01.777904 30228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:01.778105 30225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:01.778685 30220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:01.779737 30220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:01.780915 30220 hybrid_clock.cc:648] HybridClock initialized: now 1790425981780881 us; error 45 us; skew 500 ppm
Sep 26 12:33:01 dist-test-slave-3jw2 krb5kdc[30207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:01.788017 30220 init.cc:377] Logged in from keytab as kudu/127.7.99.62@KRBTEST.COM (short username kudu)
I20260926 12:33:01.789013 30220 webserver.cc:460] Webserver started at http://127.7.99.62:42577/ using document root <none> and password file <none>
I20260926 12:33:01.789602 30220 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:01.789695 30220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:01.789942 30220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:01.791975 30220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data/instance:
uuid: "b8cf473edd2e46fc93a72c393d703a02"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "f4bce718fabc05cd34c05dae980816f9"
server_key_iv: "0a2deb15048a4aec96532ce4d15051ad"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.792619 30220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance:
uuid: "b8cf473edd2e46fc93a72c393d703a02"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "f4bce718fabc05cd34c05dae980816f9"
server_key_iv: "0a2deb15048a4aec96532ce4d15051ad"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.796267 30220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:33:01.798632 30235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:01.799822 30220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:01.799973 30220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal
uuid: "b8cf473edd2e46fc93a72c393d703a02"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "f4bce718fabc05cd34c05dae980816f9"
server_key_iv: "0a2deb15048a4aec96532ce4d15051ad"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.800086 30220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:01.811241 30220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:01.814342 30220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:01.814555 30220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:01.821578 30220 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.62:46131
I20260926 12:33:01.821861 30287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.62:46131 every 8 connection(s)
I20260926 12:33:01.822573 30220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:01.823539 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 30220
I20260926 12:33:01.823657 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-0/wal/instance
I20260926 12:33:01.823982 7564 external_mini_cluster.cc:1428] Setting key de96cd32d0962fe71eea7784b2223cd3
I20260926 12:33:01.825999 30288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:01.830016 30288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.61@KRBTEST.COM" created.
Sep 26 12:33:01 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:33:01.846128 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.832337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46131 (local address 127.0.0.1:58028)
0926 12:33:01.832889 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.832945 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:01.833790 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:01.834521 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:01.834532 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:01.834850 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:01.835148 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:01.835161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:01.836600 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:01.836607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:01.837982 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:01.837997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:01.838712 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:01.838716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:01.838915 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:01.839518 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:01.839556 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:01.841324 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:01.843292 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:01.843302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:01.843316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:01.843608 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:01.843917 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:01.843921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:01.843924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:01.844036 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:01.844687 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:01.844693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:01.844839 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 12:33:01.845027 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 12:33:01.845388 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":121,"spinlock_wait_cycles":100224,"thread_start_us":222,"threads_started":1}
I20260926 12:33:01.846139 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.832337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.7.99.62:46131)
0926 12:33:01.832831 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:01.832845 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:01.833052 (+ 207us) server_negotiation.cc:407] Connection header received
0926 12:33:01.834068 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:01.834073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:01.834313 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:01.834488 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:01.835297 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:01.836474 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:01.838106 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:01.838542 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:01.838664 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:01.841473 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:01.841484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:01.841494 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:01.841514 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:01.843163 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:01.843718 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:01.843723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:01.843724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:01.843822 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:01.844140 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:01.844144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:01.844145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:01.844565 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:01.844713 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:01.845119 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 12:33:01.845308 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":373,"thread_start_us":165,"threads_started":1}
I20260926 12:33:01.848856 30288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:33:01.849581 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.849265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45122)
0926 12:33:01.849416 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.849521 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 12:33:01.849789 30238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
Entry for principal kudu/127.7.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab.
I20260926 12:33:01.850750 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.850560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45138)
0926 12:33:01.850657 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.850701 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:33:01.850912 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:39219: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
I20260926 12:33:01.851171 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:39219
--webserver_interface=127.7.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:01.900206 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } attempt: 1
I20260926 12:33:01.901113 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.900819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45146)
0926 12:33:01.900975 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.901036 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 12:33:01.901904 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.901723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45160)
0926 12:33:01.901814 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.901855 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 12:33:01.902096 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:39219: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
W20260926 12:33:01.961018 30295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:33:01.961339 30295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:01.961406 30295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:01.961484 30295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:01.965495 30295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:01.965569 30295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:01.965620 30295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:01.965675 30295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:01.965731 30295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:01.970244 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } attempt: 2
I20260926 12:33:01.971072 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.970779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45166)
0926 12:33:01.970932 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.970997 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:33:01.971221 30295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.61.keytab
--principal=kudu/127.7.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.61:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:01.971830 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:01.971599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45178)
0926 12:33:01.971727 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:01.971778 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 12:33:01.971946 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.61:39219: Network error: Client connection negotiation failed: client connection to 127.7.99.61:39219: connect: Connection refused (error 111)
I20260926 12:33:01.972396 30295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:01.973407 30295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:01.981557 30303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:01.981732 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:01.981768 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:01.981959 30295 server_base.cc:1034] running on GCE node
I20260926 12:33:01.982504 30295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:01.983659 30295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:01.984848 30295 hybrid_clock.cc:648] HybridClock initialized: now 1790425981984826 us; error 29 us; skew 500 ppm
Sep 26 12:33:01 dist-test-slave-3jw2 krb5kdc[30207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:01.987771 30295 init.cc:377] Logged in from keytab as kudu/127.7.99.61@KRBTEST.COM (short username kudu)
I20260926 12:33:01.988777 30295 webserver.cc:460] Webserver started at http://127.7.99.61:46141/ using document root <none> and password file <none>
I20260926 12:33:01.989329 30295 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:01.989398 30295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:01.989683 30295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:01.991741 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data/instance:
uuid: "1a5e7cae15424870af15ef309a6dc122"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "a874add4f647d23efde322d698da208e"
server_key_iv: "41a82dd2bfa2cd3f9b316beefabdf400"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.992373 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal/instance:
uuid: "1a5e7cae15424870af15ef309a6dc122"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "a874add4f647d23efde322d698da208e"
server_key_iv: "41a82dd2bfa2cd3f9b316beefabdf400"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.996037 30295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:01.998445 30310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:01.999574 30295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:01.999716 30295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal
uuid: "1a5e7cae15424870af15ef309a6dc122"
format_stamp: "Formatted at 2026-09-26 12:33:01 on dist-test-slave-3jw2"
server_key: "a874add4f647d23efde322d698da208e"
server_key_iv: "41a82dd2bfa2cd3f9b316beefabdf400"
server_key_version: "encryptionkey@0"
I20260926 12:33:01.999820 30295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:02.022356 30295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:02.025067 30295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:02.025264 30295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:02.032424 30295 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.61:39219
I20260926 12:33:02.032716 30362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.61:39219 every 8 connection(s)
I20260926 12:33:02.033384 30295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/data/info.pb
I20260926 12:33:02.036006 30363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:02.037493 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 30295
I20260926 12:33:02.037633 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-1/wal/instance
I20260926 12:33:02.037931 7564 external_mini_cluster.cc:1428] Setting key 825e87fedc6df814d7c908fcb2f00aa4
I20260926 12:33:02.039686 30363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.99.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.99.60@KRBTEST.COM" created.
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:33:02.054380 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.041643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.7.99.62:46131)
0926 12:33:02.041791 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.041794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.042708 (+ 914us) server_negotiation.cc:407] Connection header received
0926 12:33:02.043615 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.043618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.043678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.043787 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.044978 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.045558 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.046728 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.046967 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.047083 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.049794 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.049799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.049804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.049835 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.051959 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.052576 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.052582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.052585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.052685 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.053037 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.053043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.053047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.053311 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.053427 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.053958 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.054166 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:33:02.055138 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.042087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46131 (local address 127.0.0.1:58044)
0926 12:33:02.042554 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.042592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.043408 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.043953 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.043959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.044379 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.044821 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.044832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.045703 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.045708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.046594 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.046604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.047152 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.047158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.047360 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.047959 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.047975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.049598 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.052081 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.052090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.052107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.052471 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.052801 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.052804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.052805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.052929 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.053419 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.053424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.053647 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.053827 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.054145 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260926 12:33:02.056571 30363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
Entry for principal kudu/127.7.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab.
I20260926 12:33:02.063107 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } attempt: 3
I20260926 12:33:02.063915 7564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
/tmp/dist-test-taskT6LLMM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:44921
--webserver_interface=127.7.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.99.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:02.066354 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.057061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45180)
0926 12:33:02.057266 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.057282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.057360 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.057929 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.057931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.058052 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.058185 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.058194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.059219 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.059221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.059753 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.059760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.060483 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.060489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.060611 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.061002 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.061012 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.062317 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.064661 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.064667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.064670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.064878 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.065216 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.065221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.065225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.065319 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.065876 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.065879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.065960 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.066078 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.066198 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 12:33:02.066565 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.057205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.7.99.61:39219)
0926 12:33:02.057532 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.057546 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.057570 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:33:02.057657 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.057662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.057821 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.057923 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.058331 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.059123 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.059849 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.060329 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.060425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.062447 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.062456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.062465 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.062485 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.064568 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.064980 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.064989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.064994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.065096 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.065433 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.065436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.065437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.065677 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.065832 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.066211 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.066375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":127,"threads_started":1}
I20260926 12:33:02.069118 30363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:33:02.069903 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.069562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59364)
0926 12:33:02.069714 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.069818 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
W20260926 12:33:02.070091 30314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.070899 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.070679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59370)
0926 12:33:02.070784 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.070834 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:33:02.071313 30363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.074095 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.063588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45196)
0926 12:33:02.063750 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.063766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.063845 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.064440 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.064446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.064648 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.064785 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.064795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.065929 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.065933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.066785 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.066795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.067443 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.067450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.067586 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.068919 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.068940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.070463 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.072608 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.072610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.072612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.072803 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.073109 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.073111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.073112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.073196 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.073631 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.073633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.073733 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.073826 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.073945 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 12:33:02.074292 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.063840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.7.99.61:39219)
0926 12:33:02.064189 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.064198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.064212 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:02.064245 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.064248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.064301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.064395 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.064910 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.065781 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.066905 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.067312 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.067414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.070654 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.070660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.070664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.070707 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.072498 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.072905 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.072911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.072913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.073006 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.073286 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.073292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.073295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.073537 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.073649 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.073935 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.074071 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260926 12:33:02.074949 30288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:33:02.075671 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.075430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59380)
0926 12:33:02.075560 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.075614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:33:02.076382 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.076134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59394)
0926 12:33:02.076294 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.076335 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 12:33:02.076521 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.108067 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 1
I20260926 12:33:02.109045 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.108632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59408)
0926 12:33:02.108839 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.108929 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 12:33:02.110046 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.109707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59422)
0926 12:33:02.109873 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.109955 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 12:33:02.110203 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.120607 30363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 1
I20260926 12:33:02.121411 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.121113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59424)
0926 12:33:02.121273 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.121335 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 12:33:02.122283 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.121969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59436)
0926 12:33:02.122160 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.122226 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260926 12:33:02.122524 30363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
W20260926 12:33:02.180625 30370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:33:02.180948 30370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:02.181007 30370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:02.181084 30370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:33:02.185353 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 2
W20260926 12:33:02.185501 30370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:02.185559 30370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:02.185593 30370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:02.185626 30370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:02.185668 30370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:02.186290 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.185964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59444)
0926 12:33:02.186152 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.186215 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 12:33:02.187129 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.186847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59460)
0926 12:33:02.187020 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.187073 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 12:33:02.187335 30288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.190680 30363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 2
I20260926 12:33:02.191393 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.191120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59470)
0926 12:33:02.191249 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.191307 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 12:33:02.191768 30370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.99.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.99.62:46131,127.7.99.61:39219,127.7.99.60:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/krb5kdc/kudu_127.7.99.60.keytab
--principal=kudu/127.7.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.99.60:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:02.192233 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.191932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59474)
0926 12:33:02.192081 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.192127 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 12:33:02.192389 30363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.99.60:44921: Network error: Client connection negotiation failed: client connection to 127.7.99.60:44921: connect: Connection refused (error 111)
I20260926 12:33:02.192950 30370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:02.193951 30370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:02.202148 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:02.202239 30376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:02.202102 30377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:02.202476 30370 server_base.cc:1034] running on GCE node
I20260926 12:33:02.203061 30370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:02.204479 30370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:02.205658 30370 hybrid_clock.cc:648] HybridClock initialized: now 1790425982205630 us; error 42 us; skew 500 ppm
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425982, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:02.208436 30370 init.cc:377] Logged in from keytab as kudu/127.7.99.60@KRBTEST.COM (short username kudu)
I20260926 12:33:02.209244 30370 webserver.cc:460] Webserver started at http://127.7.99.60:41195/ using document root <none> and password file <none>
I20260926 12:33:02.209780 30370 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:02.209838 30370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:02.210014 30370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:02.211908 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data/instance:
uuid: "e455cf6d1c0c48088b0e2994a8747e08"
format_stamp: "Formatted at 2026-09-26 12:33:02 on dist-test-slave-3jw2"
server_key: "a7d05046cb336e73734c5c6312e59c68"
server_key_iv: "771f019aaf26cd0393b263fbe1e8cf86"
server_key_version: "encryptionkey@0"
I20260926 12:33:02.212597 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal/instance:
uuid: "e455cf6d1c0c48088b0e2994a8747e08"
format_stamp: "Formatted at 2026-09-26 12:33:02 on dist-test-slave-3jw2"
server_key: "a7d05046cb336e73734c5c6312e59c68"
server_key_iv: "771f019aaf26cd0393b263fbe1e8cf86"
server_key_version: "encryptionkey@0"
I20260926 12:33:02.216244 30370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 12:33:02.218541 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:02.219759 30370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:33:02.219872 30370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data,/tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal
uuid: "e455cf6d1c0c48088b0e2994a8747e08"
format_stamp: "Formatted at 2026-09-26 12:33:02 on dist-test-slave-3jw2"
server_key: "a7d05046cb336e73734c5c6312e59c68"
server_key_iv: "771f019aaf26cd0393b263fbe1e8cf86"
server_key_version: "encryptionkey@0"
I20260926 12:33:02.219962 30370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:02.231560 30370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:02.234836 30370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:02.235001 30370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:02.242360 30370 rpc_server.cc:307] RPC server started. Bound to: 127.7.99.60:44921
I20260926 12:33:02.242645 30438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.99.60:44921 every 8 connection(s)
I20260926 12:33:02.243364 30370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/data/info.pb
I20260926 12:33:02.246176 30439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:02.249869 30439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:33:02.250725 7564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu as pid 30370
I20260926 12:33:02.250873 7564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT6LLMM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425700812969-7564-0/minicluster-data/master-2/wal/instance
I20260926 12:33:02.251252 7564 external_mini_cluster.cc:1428] Setting key 8dfa7a6ce11944595966764938cfb642
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425982, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:33:02.263432 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.251874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.7.99.62:46131)
0926 12:33:02.252041 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.252046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.252930 (+ 884us) server_negotiation.cc:407] Connection header received
0926 12:33:02.253857 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.253861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.253923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.254016 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.254866 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.255369 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.256437 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.256654 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.256767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.259580 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.259587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.259594 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.259632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.261371 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.261919 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.261922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.261924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.262012 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.262348 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.262353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.262355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.262537 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.262645 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.263124 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.263254 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:33:02.264340 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.252316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.62:46131 (local address 127.0.0.1:58058)
0926 12:33:02.252787 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.252828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.253650 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.254069 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.254079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.254428 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.254720 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.254732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.255480 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.255485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.256316 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.256325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.256800 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.256804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.256983 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.257642 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.257667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.259414 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.261500 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.261509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.261518 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.261815 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.262112 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.262115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.262117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.262254 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.262642 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.262649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.262898 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.263070 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.263370 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260926 12:33:02.264348 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.253050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.7.99.60:44921)
0926 12:33:02.253331 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.253345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.253379 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:33:02.253667 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.253687 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.253887 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.254146 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.255052 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.255894 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.256823 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.257215 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.257334 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.260042 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.260048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.260058 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.260081 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.261770 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.262229 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.262237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.262245 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.262327 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.262646 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.262649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.262651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.262912 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.263148 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.263350 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.263499 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260926 12:33:02.266300 30439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425982, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.62@KRBTEST.COM
I20260926 12:33:02.275233 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.266808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.61:39219 (local address 127.0.0.1:45202)
0926 12:33:02.267031 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.267046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.267129 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.267394 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.267397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.267573 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.267660 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.267665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.268501 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.268506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.269020 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.269027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.269505 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.269509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.269603 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.270106 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.270124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.271704 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.273747 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.273753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.273756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.273955 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.274247 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.274250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.274253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.274338 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.274723 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.274726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.274822 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.274926 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.275057 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 12:33:02.275249 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.266901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.7.99.61:39219)
0926 12:33:02.267033 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.267038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.267088 (+ 50us) server_negotiation.cc:407] Connection header received
0926 12:33:02.267240 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.267243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.267299 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.267384 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.267787 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.268395 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.269140 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.269393 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.269505 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.271874 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.271877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.271879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.271905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.273635 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.274055 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.274060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.274062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.274147 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.274434 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.274437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.274439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.274624 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.274730 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.274986 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.275103 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:33:02.275866 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.266920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.7.99.62:46131)
0926 12:33:02.267034 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.267039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.267137 (+ 98us) server_negotiation.cc:407] Connection header received
0926 12:33:02.267260 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.267265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.267314 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.267404 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.268027 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.268577 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.269359 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.269583 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.269701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.272675 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.272686 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.272690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.272714 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.274091 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.274537 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.274542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.274544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.274635 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.274941 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.274945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.274946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.275171 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.275416 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.275553 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.275671 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 12:33:02.276353 30439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425982, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.60@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.99.61@KRBTEST.COM
I20260926 12:33:02.283556 30363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 3
I20260926 12:33:02.285111 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.276902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59486)
0926 12:33:02.277058 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.277073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.277141 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.277463 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.277466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.277640 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.277736 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.277747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.278524 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.278527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.279160 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.279171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.279609 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.279614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.279698 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.280213 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.280231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.281531 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.283401 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.283405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.283407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.283635 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.283933 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.283936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.283938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.284040 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.284527 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.284530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.284633 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.284763 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.284902 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 12:33:02.285683 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.276983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.7.99.60:44921)
0926 12:33:02.277090 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.277095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.277118 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:33:02.277223 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.277226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.277273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.277352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.277863 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.278410 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.279286 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.279506 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.279576 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.281662 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.281665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.281668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.281696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.283299 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.283756 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.283759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.283762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.283839 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.284134 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.284138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.284141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.284393 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.284504 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.285369 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.285494 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:33:02.286516 30288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } attempt: 3
I20260926 12:33:02.287616 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.277386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.7.99.61:39219)
0926 12:33:02.277495 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.277498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.277510 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:02.277545 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.277547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.277578 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.277637 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.278309 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.278879 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.279833 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.280226 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.280316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.282826 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.282831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.282833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.282861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.285078 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.285598 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.285602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.285604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.285715 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.286071 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.286074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.286076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.286285 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.286487 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.287301 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.287429 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:33:02.289633 30439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: Bootstrap starting.
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.61@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:33:02.292045 7564 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 26 12:33:02 dist-test-slave-3jw2 krb5kdc[30207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425981, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.99.62@KRBTEST.COM for kudu/127.7.99.60@KRBTEST.COM
I20260926 12:33:02.293401 30439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:02.293516 30262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b8cf473edd2e46fc93a72c393d703a02"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58074
I20260926 12:33:02.294910 30439 log.cc:824] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:02.295543 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.283979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59498)
0926 12:33:02.284132 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.284148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.284242 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.284755 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.284758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.285031 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.285106 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.285112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.287180 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.287183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.287918 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.287928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.288595 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.288599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.288675 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.289105 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.289121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.290865 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.293729 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.293735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.293741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.294004 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.294344 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.294348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.294350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.294446 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.294981 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.294984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.295095 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.295186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.295350 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:33:02.296192 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.284241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.7.99.60:44921)
0926 12:33:02.284557 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.284565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.284575 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:33:02.284609 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.284611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.284641 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.284741 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.285414 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.287043 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.288124 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.288468 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.288897 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.291222 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.291227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.291229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.291254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.293574 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.294110 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.294113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.294115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.294225 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.294584 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.294587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.294588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.294850 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.295742 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.295827 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.295955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260926 12:33:02.297091 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.286931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.99.60:44921 (local address 127.0.0.1:59510)
0926 12:33:02.287254 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:02.287270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:02.287353 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:02.288053 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:02.288057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:02.288283 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:02.288419 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.288428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.289380 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.289385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.290278 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:02.290293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.290968 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.290971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:02.291066 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.291531 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:02.291545 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:02.293044 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:02.295434 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.295439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.295441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.295668 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.295977 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:02.295981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:02.295983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:02.296071 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:02.296542 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:02.296547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:02.296661 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 12:33:02.296794 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 12:33:02.296927 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260926 12:33:02.297325 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.287514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.7.99.60:44921)
0926 12:33:02.287616 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.287620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.287635 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:02.287875 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.287878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.287917 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.288031 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.288580 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.289232 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.290441 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.290785 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.290954 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.293361 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.293364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.293368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.293394 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.295330 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.295797 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.295800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.295802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.295881 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.296201 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.296206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.296208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.296425 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.296523 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.296884 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.297067 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:02.298485 30439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: No bootstrap required, opened a new log
I20260926 12:33:02.299222 30363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: Bootstrap starting.
I20260926 12:33:02.299974 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: Bootstrap starting.
I20260926 12:33:02.301936 30363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:02.302443 30288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:02.302841 30363 log.cc:824] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:02.303004 30439 raft_consensus.cc:348] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.303272 30439 raft_consensus.cc:729] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e455cf6d1c0c48088b0e2994a8747e08, State: Initialized, Role: FOLLOWER
I20260926 12:33:02.303371 30288 log.cc:824] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:02.303613 30439 consensus_queue.cc:260] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.304132 30448 sys_catalog.cc:455] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.304375 30448 sys_catalog.cc:458] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.305047 30439 sys_catalog.cc:564] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:02.305500 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: No bootstrap required, opened a new log
I20260926 12:33:02.308810 30288 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.309042 30288 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cf473edd2e46fc93a72c393d703a02, State: Initialized, Role: FOLLOWER
I20260926 12:33:02.309216 30363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: No bootstrap required, opened a new log
I20260926 12:33:02.309368 30288 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
W20260926 12:33:02.310201 30459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:02.310267 30459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:02.310491 30461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.310607 30461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.310653 30288 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:02.312491 30262 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:33:02.308840 30370 master_runner.cc:186] Error getting master registration for 127.7.99.62:46131: OK, instance_id { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" instance_seqno: 1790425981819665 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:33:02.312610 30262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:02.312685 30370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:33:02.313553 30363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.313788 30363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5e7cae15424870af15ef309a6dc122, State: Initialized, Role: FOLLOWER
I20260926 12:33:02.313907 30262 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.314131 30363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.314611 30467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.314772 30467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.314869 30262 leader_election.cc:288] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [CANDIDATE]: Term 1 election: Requested vote from peers 1a5e7cae15424870af15ef309a6dc122 (127.7.99.61:39219), e455cf6d1c0c48088b0e2994a8747e08 (127.7.99.60:44921)
I20260926 12:33:02.315361 30413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8cf473edd2e46fc93a72c393d703a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e455cf6d1c0c48088b0e2994a8747e08"
I20260926 12:33:02.315634 30413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:33:02.316018 30241 tablet.cc:2369] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:02.317008 30413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cf473edd2e46fc93a72c393d703a02 in term 1.
I20260926 12:33:02.317840 30363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:02.317893 30239 leader_election.cc:302] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cf473edd2e46fc93a72c393d703a02, e455cf6d1c0c48088b0e2994a8747e08; no voters:
I20260926 12:33:02.318179 30461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:02.318135 30337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8cf473edd2e46fc93a72c393d703a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1a5e7cae15424870af15ef309a6dc122"
I20260926 12:33:02.318361 30337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:02.318408 30461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [term 1 LEADER]: Becoming Leader. State: Replica: b8cf473edd2e46fc93a72c393d703a02, State: Running, Role: LEADER
I20260926 12:33:02.318727 30461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } }
I20260926 12:33:02.319811 30337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cf473edd2e46fc93a72c393d703a02 in term 1.
I20260926 12:33:02.320570 30461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8cf473edd2e46fc93a72c393d703a02. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.320668 30461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:02.326226 30491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 12:33:02.327736 30489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:02.327823 30489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:02.333344 30491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:02.345956 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.325871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.7.99.61:39219)
0926 12:33:02.325978 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.325982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.333244 (+ 7262us) server_negotiation.cc:407] Connection header received
0926 12:33:02.333423 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.333428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.333495 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.333557 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.334547 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.335311 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.336565 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.336903 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.336989 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.339647 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.339651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.339654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.339682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.341490 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.344267 (+ 2777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.344272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.344274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.344378 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.344790 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.344796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.344800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.345053 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.345192 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.345597 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.345726 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:33:02.347625 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.325442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.7.99.60:44921)
0926 12:33:02.325565 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.325570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.332955 (+ 7385us) server_negotiation.cc:407] Connection header received
0926 12:33:02.333025 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.333028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.333081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.333162 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.337602 (+ 4440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.338389 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.340546 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.340912 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.341022 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.342892 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.342898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.342901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.342931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.345398 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.345908 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.345912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.345913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.346020 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.346599 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.346603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.346604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.346832 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.346947 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.347326 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.347450 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:33:02.351121 30413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [term 1 FOLLOWER]: Refusing update from remote peer b8cf473edd2e46fc93a72c393d703a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:02.351855 30478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:02.355082 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:02.323651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.7.99.62:46131)
0926 12:33:02.323777 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.323782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.332526 (+ 8744us) server_negotiation.cc:407] Connection header received
0926 12:33:02.332684 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.332688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.332745 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.332825 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.338537 (+ 5712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.339365 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.343358 (+ 3993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:02.343692 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:02.343803 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:02.351190 (+ 7387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:02.351196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:02.351202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:02.351235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:02.353080 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.353614 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.353618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.353620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.353711 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:02.354037 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:02.354040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:02.354042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:02.354280 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:02.354399 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:02.354753 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 12:33:02.354888 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:02.356276 30448 sys_catalog.cc:455] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8cf473edd2e46fc93a72c393d703a02. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.356427 30448 sys_catalog.cc:458] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.356845 30337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [term 1 FOLLOWER]: Refusing update from remote peer b8cf473edd2e46fc93a72c393d703a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:02.357443 30478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:02.361056 30467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8cf473edd2e46fc93a72c393d703a02. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.361181 30467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.365571 30478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.365701 30478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:02.366032 30497 mvcc.cc:205] Tried to move back new op lower bound from 7333584823701696512 to 7333584823580876800. Current Snapshot: MvccSnapshot[applied={T|T < 7333584823701696512}]
I20260926 12:33:02.367142 30448 sys_catalog.cc:455] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.367252 30448 sys_catalog.cc:458] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.367743 30498 mvcc.cc:205] Tried to move back new op lower bound from 7333584823701696512 to 7333584823580876800. Current Snapshot: MvccSnapshot[applied={T|T < 7333584823701696512}]
I20260926 12:33:02.371632 30461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.371762 30461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:02.372833 30467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8cf473edd2e46fc93a72c393d703a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cf473edd2e46fc93a72c393d703a02" member_type: VOTER last_known_addr { host: "127.7.99.62" port: 46131 } } peers { permanent_uuid: "1a5e7cae15424870af15ef309a6dc122" member_type: VOTER last_known_addr { host: "127.7.99.61" port: 39219 } } peers { permanent_uuid: "e455cf6d1c0c48088b0e2994a8747e08" member_type: VOTER last_known_addr { host: "127.7.99.60" port: 44921 } } }
I20260926 12:33:02.372938 30467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:02.373965 30500 mvcc.cc:205] Tried to move back new op lower bound from 7333584823701696512 to 7333584823580876800. Current Snapshot: MvccSnapshot[applied={T|T < 7333584823701696512}]
I20260926 12:33:02.376580 30491 catalog_manager.cc:1347] Generated new cluster ID: 596fa76c97994f09b3b762882cf9d0d3
I20260926 12:33:02.376655 30491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:02.398875 30491 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:02.400225 30491 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:02.414597 30491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8cf473edd2e46fc93a72c393d703a02: Generated new TSK 0
I20260926 12:33:02.415567 30491 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:33:02.443084 30291 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.439458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.7.99.62:46131)
0926 12:33:02.439584 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.439589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.441781 (+ 2192us) server_negotiation.cc:407] Connection header received
0926 12:33:02.442190 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.442194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.442249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.442308 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.442945 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58094: BlockingRecv error: recv got EOF from 127.0.0.1:58094 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:33:02.444581 30443 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.440181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.7.99.60:44921)
0926 12:33:02.440317 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.440322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.442108 (+ 1786us) server_negotiation.cc:407] Connection header received
0926 12:33:02.443794 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.443797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.443846 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.443897 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.444469 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59524: BlockingRecv error: recv got EOF from 127.0.0.1:59524 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 12:33:02.445756 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.441120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.7.99.61:39219)
0926 12:33:02.441240 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.441244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.443711 (+ 2467us) server_negotiation.cc:407] Connection header received
0926 12:33:02.445024 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.445028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.445075 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.445122 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.445617 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45220: BlockingRecv error: recv got EOF from 127.0.0.1:45220 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:33:02.450340 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.448854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.7.99.61:39219)
0926 12:33:02.448948 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.448952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.449407 (+ 455us) server_negotiation.cc:407] Connection header received
0926 12:33:02.449618 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.449622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.449714 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.449784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.450249 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45234: BlockingRecv error: recv got EOF from 127.0.0.1:45234 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 12:33:02.455058 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.453668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.7.99.61:39219)
0926 12:33:02.453775 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.453778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.454260 (+ 482us) server_negotiation.cc:407] Connection header received
0926 12:33:02.454424 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.454428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.454472 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.454519 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.454971 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45244: BlockingRecv error: recv got EOF from 127.0.0.1:45244 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 12:33:02.460469 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.458980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.7.99.61:39219)
0926 12:33:02.459066 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.459073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.459545 (+ 472us) server_negotiation.cc:407] Connection header received
0926 12:33:02.459718 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.459721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.459763 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.459809 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.460382 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45260: BlockingRecv error: recv got EOF from 127.0.0.1:45260 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 12:33:02.468051 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.466467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.7.99.61:39219)
0926 12:33:02.466600 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.466604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.467177 (+ 573us) server_negotiation.cc:407] Connection header received
0926 12:33:02.467373 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.467377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.467425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.467474 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.467947 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45272: BlockingRecv error: recv got EOF from 127.0.0.1:45272 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 12:33:02.480578 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.478615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.7.99.61:39219)
0926 12:33:02.478741 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.478746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.479376 (+ 630us) server_negotiation.cc:407] Connection header received
0926 12:33:02.479603 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.479607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.479679 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.479740 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.480446 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45276: BlockingRecv error: recv got EOF from 127.0.0.1:45276 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:33:02.501381 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.499399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.7.99.61:39219)
0926 12:33:02.499565 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.499570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.500248 (+ 678us) server_negotiation.cc:407] Connection header received
0926 12:33:02.500517 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.500521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.500585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.500649 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.501257 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45280: BlockingRecv error: recv got EOF from 127.0.0.1:45280 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:33:02.537454 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.536089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.7.99.61:39219)
0926 12:33:02.536239 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.536244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.536518 (+ 274us) server_negotiation.cc:407] Connection header received
0926 12:33:02.536698 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.536702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.536752 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.536844 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.537324 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45284: BlockingRecv error: recv got EOF from 127.0.0.1:45284 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:33:02.605276 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.603951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.7.99.61:39219)
0926 12:33:02.604122 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.604127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.604350 (+ 223us) server_negotiation.cc:407] Connection header received
0926 12:33:02.604524 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.604531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.604591 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.604693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.605149 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45298: BlockingRecv error: recv got EOF from 127.0.0.1:45298 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 12:33:02.736591 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.735329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.7.99.61:39219)
0926 12:33:02.735506 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.735511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.735651 (+ 140us) server_negotiation.cc:407] Connection header received
0926 12:33:02.735847 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.735851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.735899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.735990 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.736466 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45306: BlockingRecv error: recv got EOF from 127.0.0.1:45306 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 12:33:02.995852 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:02.994697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.7.99.61:39219)
0926 12:33:02.994876 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:02.994881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:02.994899 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:02.995074 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:02.995077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:02.995130 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:02.995207 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:02.995723 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45318: BlockingRecv error: recv got EOF from 127.0.0.1:45318 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 12:33:03.311491 30459 catalog_manager.cc:1259] Loaded cluster ID: 596fa76c97994f09b3b762882cf9d0d3
I20260926 12:33:03.311610 30459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: loading cluster ID for follower catalog manager: success
I20260926 12:33:03.313419 30459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: acquiring CA information for follower catalog manager: success
I20260926 12:33:03.314525 30459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e455cf6d1c0c48088b0e2994a8747e08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:33:03.329017 30489 catalog_manager.cc:1259] Loaded cluster ID: 596fa76c97994f09b3b762882cf9d0d3
I20260926 12:33:03.329082 30489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: loading cluster ID for follower catalog manager: success
I20260926 12:33:03.330694 30489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: acquiring CA information for follower catalog manager: success
I20260926 12:33:03.331811 30489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a5e7cae15424870af15ef309a6dc122: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:33:03.513880 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:03.509856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.7.99.61:39219)
0926 12:33:03.510221 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:03.510225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:03.510246 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:33:03.510334 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:03.510336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:03.510466 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:03.510569 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:03.510800 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.511499 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.512423 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.512648 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.512728 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:03.512993 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:03.513467 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:03.513568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:03.513618 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 12:33:03.513671 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
I20260926 12:33:03.520540 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:03.516811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.7.99.60:44921)
0926 12:33:03.517049 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:03.517052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:03.517065 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:03.517138 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:03.517140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:03.517245 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:03.517341 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:03.517628 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.518166 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.519024 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.519254 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.519347 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:03.519647 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:03.520118 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:03.520293 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:03.520362 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 12:33:03.520413 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260926 12:33:03.527401 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:03.523645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.7.99.61:39219)
0926 12:33:03.523797 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:03.523802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:03.523917 (+ 115us) server_negotiation.cc:407] Connection header received
0926 12:33:03.524052 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:03.524055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:03.524107 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:03.524215 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:03.524430 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.525159 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.526256 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.526597 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.526702 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:03.526968 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:03.527055 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:03.527119 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:03.527211 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:33:03.527262 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 12:33:03.528093 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:03.523875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.7.99.60:44921)
0926 12:33:03.523978 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:03.523995 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:03.524620 (+ 625us) server_negotiation.cc:407] Connection header received
0926 12:33:03.524707 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:03.524710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:03.524758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:03.524831 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:03.525089 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.525756 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.526870 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.527203 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.527296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:03.527602 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:03.527670 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:03.527717 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:03.527897 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 12:33:03.527952 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:33:03.528618 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:03.523423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.7.99.62:46131)
0926 12:33:03.523816 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:03.523822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:03.523836 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:03.523929 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:03.523933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:03.524096 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:03.524539 (+ 443us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:03.524678 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.525465 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.526556 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:03.526898 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:03.526998 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:03.527326 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:03.527692 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:03.527785 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:03.528381 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 12:33:03.528453 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":83,"threads_started":1}
I20260926 12:33:03.532409 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 30220
I20260926 12:33:03.538748 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 30295
I20260926 12:33:03.544430 7564 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT6LLMM/build/debug/bin/kudu with pid 30370
2026-09-26T12:33:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3068 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6193 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282724 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST